Deep Learning Models For Tumor Evaluation

ABSTRACT

A method of determining a clinical value for an individual based on a tumor in an image by an apparatus including processing circuitry may include executing, by the processing circuitry, instructions that cause the apparatus to determine a lymphocyte distribution of lymphocytes in the tumor based on the image; apply a classifier to the lymphocyte distribution to classify the tumor, the classifier having been trained to classify tumors into a class selected from at least two classes respectively associated with lymphocyte distributions; and determine the clinical value for the individual based on prognoses of individuals with tumors in the class into which the classifier classified the tumor.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/959,931 filed Jan. 11, 2020, the entire disclosure of which isincorporated by reference.

FIELD

The present disclosure relates to the field of image analysis usingmachine learning models, and more particularly to determining clinicalvariables related to tumors using deep learning models.

BACKGROUND

The background description provided here is for the purpose of generallypresenting the context of the disclosure. Work of the presently namedinventors, to the extent it is described in this background section, aswell as aspects of the description that may not otherwise qualify asprior art at the time of filing, are neither expressly nor impliedlyadmitted as prior art against the present disclosure.

In the field of medicine, many scenarios involve an analysis of cancertumors to evaluate a tumor class based on characteristics such aslocation, size, shape, and composition. The evaluation may enablepredictions such as the tumor behavior and likely aggressiveness, suchas the probability and rate of growth and/or metastasis. Theseproperties of the tumor may in turn enable determinations about theclinical value (such as the prognosis) for the individual, such as thelikely survival rate, and may guide decisions about medical treatment,such as a selection, type, and/or timing of chemotherapy, surgery, andpalliative care. However, the determination of prognosis, includingsurvivability, is difficult to the number and variety of relevantfactors and factor correlations that may affect this determination.

A wide variety of diagnostic and prognostic techniques may be used toperform the evaluation of tumor classes. For example, a collection ofdata may include features about individuals with tumors, such as eachindividual's age, physiology, medical history, and/or behaviors such assmoking, may be correlated with prognostic data of the individual, suchas typical survival rates. A Cox proportional hazards models may beapplied to determine the correlation of features of a clinical featureset based on the collected data set and the clinical data, which maysupport some conclusions about the relevance of respective risk factorsfor the clinical value (such as the prognosis) for the individual basedon the tumor. Clinicians may thereafter use this information to guidedeterminations or predictions about the diagnosis, prognosis, and/oreffective care options for individuals with similar tumors. Also,similar risk factors may be collected about such individuals andprocessed through the Cox proportional hazards model to predict theclinical value (such as the prognosis) for the individual based onsimilar tumors upon which the Cox proportional hazards model wasdeveloped.

Other techniques for evaluating the clinical value (such as theprognosis) for an individual based on a tumor may utilize one or moremachine learning models. For example, a training data set of tumorsamples with known properties, such as data about the tumor, theindividual from whom the tumor were removed, and/or the clinical value(such as the prognosis) for the individual, may be generated. A machinelearning classifier may be trained using the training data set withlabels that indicate the classes represented by each input, such aswhether each tumor represents a high-risk tumor with a poor prognosis ora low-risk tumor with a good prognosis. The training process may resultin a trained machine learning classifier that classifies new input in amanner that is consistent with the examples of the training data set.

A variety of different machine learning models may be selected asclassifiers for tumors, such as Bayesian classifiers, artificial neuralnetworks, and support vector machines (SVMs). As a first such example, aconvolutional neural network (CNN) may process an n-dimensional input todetect features of images of tumors that may occur therein. A featurevector, such as a pixel array of an image of a tumor, may be provided toa convolutional neural network including a sequence of convolutionallayers of neurons. Each convolutional layer may produce a feature mapindicating some image features of the tumor that were detected at alevel of detail, which may be processed by a next convolutional layer inthe sequence. The feature map produced by the final convolutional layerof the convolutional neural network may be processed by a classifier ofthe tumor, which may be trained to indicate whether the feature map issimilar to the feature maps of objects in the images of the trainingdata set. For example, the CNN may be trained to identify visualfeatures of a tumor that are correlated with high-risk and low-riskprognoses.

As a second such example, a Gaussian Mixture Model (GMM) may begenerated to classify data about tumors into different clusters oftumors with representative properties. For each sample in the trainingdata set representing a tumor, a set of features may be identified, suchas location, size, shape, and composition. The samples of the trainingdata set may be positioned within a multidimensional feature space,where each feature is represented along a dimensional axis. Machinelearning techniques may be applied to identify clusters of tumors withinthe feature space that share a similar prognosis, such as a firstcluster representing high-risk tumors and second cluster representinglow-risk tumors, where each cluster is represented as a collection ofGaussian probability distributions of the respective features within thefeature space. Even if some parts of the clusters overlap (for example,even if a tumor with particular set of features could be included ineither the high-risk cluster or the low-risk cluster), the Gaussianprobability distributions of the clusters may enable a probabilisticprediction as to the likelihood of the tumor belonging to each cluster.In this manner, the Gaussian mixture model may enable individualprognosis prediction based on clustering of similar tumor samples in thetraining data set.

BRIEF SUMMARY

Some example embodiments may include a method of operating an apparatusincluding processing circuitry, in which the method includes executing,by the processing circuitry, instructions that cause the apparatus toreceive an image depicting at least part of a tumor, determine alymphocyte distribution of lymphocytes in the tumor based on the image,apply a classifier to the lymphocyte distribution to classify the tumor,the classifier having been trained to classify tumors into a classselected from at least two classes respectively associated withlymphocyte distributions, and determine a clinical value for anindividual based on a set of prognosis data corresponding to individualswith tumors in the class into which the classifier classified the tumor.

In some example embodiments, the tumor is a pancreatic adenocarcinomatumor or a breast cancer tumor.

In some example embodiments, the apparatus may further include aconvolutional neural network that is trained to determine a lymphocytedistribution of lymphocytes in an area of an image, and the instructionsmay cause the apparatus to invoke the convolutional neural network todetermine the lymphocyte distribution of lymphocytes in respective areasof the image of the tumor. In some example embodiments, theconvolutional neural network may be further trained to classify an areaof the image as one or more area types selected from an area type setincluding, a tumor area, a lymphocyte area, or a stroma area. In someexample embodiments, determining the lymphocyte distribution oflymphocytes in the tumor may include, for respective lymphocyte areas ofthe image and determining a distance of the lymphocyte area to one orboth of a tumor area or a stroma area, based on the distance,characterizing the lymphocyte area as one of, a tumor-infiltratinglymphocyte area, a tumor-adjacent lymphocyte area, a stroma-infiltratinglymphocyte area, or a stroma-adjacent lymphocyte area, and theclassifier may further classify the tumor based on the characterizing ofthe lymphocyte area. In some example methods, determining the lymphocytedistribution of lymphocytes in the tumor may include, for respectivestroma areas of the image, determining a distance of the stroma area toa tumor area, and based on the distance, characterizing the stroma areaas one of, a tumor-infiltrating stroma area, or a tumor-adjacent stromaarea, and the classifier may further classify the tumor based on thecharacterizing of the stroma area.

In some example embodiments, the at least two classes may include, ahigh-risk class of tumors that are associated with a first survivalprobability, and a low-risk class of tumors that are associated with asecond survival probability that is longer than the first survivalprobability.

In some example embodiments, the classifier may further include aGaussian mixture model configured to determine, for respective classes,a probability distribution of features for tumors in the class within afeature space. In some example embodiments, the features of the featurespace of the Gaussian mixture model may be selected from a feature setincluding, a measurement of tumor areas of the image, a measurement ofstroma areas of the image, a measurement of lymphocyte areas of theimage, a measurement of tumor-infiltrating lymphocyte areas of theimage, a measurement of tumor-adjacent lymphocyte areas of the image, ameasurement of stroma-infiltrating lymphocyte areas of the image, ameasurement of stroma-adjacent lymphocyte areas of the image, ameasurement of tumor-infiltrating stroma areas of the image, and ameasurement of tumor-adjacent stroma areas of the image. In some exampleembodiments, from the feature set, a feature subset may be selectedbased on a correlation of the respective classes with respectivefeatures of the subset. In some example embodiments, the correlation ofthe respective classes with the respective features may be based on oneor both of, a silhouette score of the feature space, or a concordanceindex. In some example embodiments, the feature subset may consistessentially of, the measurement of lymphocyte areas of the image, themeasurement of tumor-infiltrating lymphocyte areas of the image, themeasurement of tumor-adjacent lymphocyte areas of the image, and themeasurement of tumor-infiltrating stroma areas of the image.

In some example embodiments, the instructions may further cause theapparatus to, apply a Cox proportional hazards model to clinicalfeatures of the tumor to determine a class of the tumor, and determinethe clinical value (such as the prognosis) for the individual based onthe prognoses for the individuals with tumors in the class into whichthe classifier classified the tumor and the class determined by the Coxproportional hazards model. In some example embodiments, the clinicalfeatures of the tumor of the Cox proportional hazards model may beselected from a clinical feature set including a primary diagnosis ofthe tumor, a location of the tumor, a treatment of the tumor, ameasurement of the tumor, a metastatic condition of the tumor, a primarydiagnosis for the individual, a previous cancer medical history of theindividual, a race of the individual, an ethnicity of the individual, agender of the individual, a smoking habit frequency of the individual, asmoking habit duration of the individual, an alcohol history of theindividual. In some example embodiments, from the clinical feature set,a clinical feature subset of clinical features may be selected for theCox proportional hazards model based on a correlation of the respectiveclasses with respective clinical features of the subset. In some exampleembodiments, the clinical feature subset may consist of, the measurementof the tumor, and the metastatic condition of the tumor.

In some example embodiments, the instructions may further cause theapparatus to display a visualization of a clinical value (such as aprognosis) for the individual. In some example embodiments, thevisualization is a Kaplan Meier survivability projection of the tumor.In some example embodiments, the instructions may further cause theapparatus to determine a diagnostic test for the tumor based on theclinical value (such as the prognosis) for the individual. In someexample embodiments, the instructions may further cause the apparatus todetermine a treatment of the individual based on the clinical value(such as the prognosis) for the individual. In some example embodiments,the instructions may further cause the apparatus to determine a scheduleof a therapeutic agent for treating the tumor based on the clinicalvalue (such as the prognosis) for the individual.

In some example embodiments, the at least two classes are a low-risktumor class and a high-risk tumor class, determining the lymphocytedistribution further includes applying a convolutional neural network tothe image, the convolutional neural network configured to measure thelymphocyte distribution of lymphocytes for different area types of theimage, the classifier is a two-way Gaussian mixture model configured todetermine, for respective classes, a probability distribution offeatures for tumors in the class within a feature space, the methodfurther includes applying a Cox proportional hazards model to clinicalfeatures of the tumor to determine the class of the tumor, anddetermining the clinical value (such as the prognosis) for theindividual is further based on the class predicted by the Coxproportional hazards model.

Some example embodiments may include a system including memory hardwareconfigured to store instructions that embody any of the above methods,and processing hardware configured to execute the instructions stored bythe memory hardware.

Some example embodiments may include a system including an imageevaluator configured to determine a lymphocyte distribution oflymphocytes in an image, a classifier configured to classify tumors intoa class selected from at least two classes respectively associated withlymphocyte distributions and a tumor evaluator configured to determine aclinical value (such as a prognosis and/or survivability) for anindividual based on a tumor in an image by, invoking the image evaluatorwith the image to determine the lymphocyte distribution of lymphocytesin the tumor, invoking the classifier to classify the tumor into a classbased on the lymphocyte distribution, and outputting a clinical value(such as a prognosis) for the individual based on prognoses ofindividuals with tumors in the class into which the classifierclassified the tumor. In some example embodiments, the at least twoclasses are a low-risk tumor class and a high-risk tumor class, theimage evaluator is a convolutional neural network configured to measurethe lymphocyte distribution of lymphocytes for different area types ofthe image, the classifier is a two-way Gaussian mixture model configuredto determine, for respective classes, a probability distribution offeatures for tumors in the class within a feature space, the systemfurther includes a Cox proportional hazards model to clinical featuresof the tumor to determine a class of the tumor, and the tumor evaluatoris further configured to determine the clinical value (such as theprognosis) for the individual based on the prognoses of the individualswith tumors in the class into which the classifier classified the tumorand the class determined by the Cox proportional hazards model.

Some example embodiments may include a system including image evaluatingmeans for determining a lymphocyte distribution of lymphocytes in animage, classifying means for classifying tumors into a class selectedfrom at least two classes respectively associated with lymphocytedistributions, and tumor evaluator means for determining a clinicalvalue (such as a prognosis) for an individual based on a tumor in animage by, invoking the image evaluating means with the image todetermine the lymphocyte distribution of lymphocytes in the tumor,invoking the classifier to classify the tumor into a class based on thelymphocyte distribution, and outputting a clinical value (such as aprognosis) for the individual based on prognoses of individuals withtumors in the class into which the classifier classified the tumor.

Some example embodiments may include an apparatus including a memorystoring instructions, and processing circuitry configured by executionof the instructions stored in the memory to determine a clinical value(such as a prognosis) for an individual based on a tumor in an image by,determining a lymphocyte distribution of lymphocytes in a tumor based onan image of the tumor, applying a classifier to the lymphocytedistribution to classify the tumor, the classifier configured toclassify tumors into a class selected from at least two classesrespectively associated with lymphocyte distributions, and outputting aclinical value (such as a prognosis) for the individual based onprognoses of individuals with tumors in the class into which theclassifier classified the tumor. In some example embodiments, the atleast two classes are a low-risk tumor class and a high-risk tumorclass, determining the lymphocyte distribution further includes applyinga convolutional neural network to the image, the convolutional neuralnetwork configured to measure the lymphocyte distribution of lymphocytesfor different area types of the image, the classifier includes a two-wayGaussian mixture model configured to determine, for respective classes,a probability distribution of features for tumors in the class within afeature space, and the instructions further cause the processingcircuitry to apply a Cox proportional hazards model to clinical featuresof the tumor to determine a class of the tumor, and determine theclinical value (such as the prognosis) for the individual based on theprognoses of the individuals with tumors in the class into which theclassifier classified the tumor and the class determined by the Coxproportional hazards model.

Some example embodiments may include a non-transitory computer-readablemedium storing instructions that, when executed by processing circuitry,cause the processing circuitry to determine a clinical value (such as aprognosis) for an individual based on a tumor in an image by,determining a lymphocyte distribution of lymphocytes in a tumor based onan image of the tumor, applying a classifier to the lymphocytedistribution to classify the tumor, the classifier configured toclassify tumors into a class selected from at least two classesrespectively associated with lymphocyte distributions, and outputting aclinical value (such as a prognosis) for the individual based onprognoses of individuals with tumors in the class into which theclassifier classified the tumor. In some example embodiments, the atleast two classes are a low-risk tumor class and a high-risk tumorclass, determining the lymphocyte distribution further includes applyinga convolutional neural network to the image, the convolutional neuralnetwork configured to measure the lymphocyte distribution of lymphocytesfor different area types of the image, the classifier includes a two-wayGaussian mixture model configured to determine, for respective classes,a probability distribution of features for tumors in the class within afeature space, and the instructions further cause the processingcircuitry to apply a Cox proportional hazards model to clinical featuresof the tumor to determine a class of the tumor, and determine theclinical value (such as the prognosis) for the individual based on theprognoses of the individuals with tumors in the class into which theclassifier classified the tumor and the class determined by the Coxproportional hazards model.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will become more fully understood from thedetailed description and the accompanying drawings. In the drawings,reference numbers may be reused to identify similar and/or identicalelements.

FIG. 1 is an illustration of an example convolutional neural network.

FIG. 2A is an illustration of an example image analysis to identify areatypes and distributions of lymphocytes in an image of a tumor inaccordance with some example embodiments.

FIG. 2B is an illustration of an example image analysis to classify thedistribution of lymphocytes in an image of a tumor in accordance withsome example embodiments.

FIG. 3 is an illustration of a set of masks of lung tissue samplesincluding a lymphocyte distribution of lymphocytes by an example machinelearning model in accordance with some example embodiments.

FIG. 4 is an illustration of an example machine learning model thatclassifies tumors in accordance with some example embodiments.

FIG. 5 is an illustration of a characterization of a set of images ofpancreatic adenocarcinoma tissue samples in accordance with some exampleembodiments.

FIG. 6A is an illustration of a set of samples arranged in atwo-dimensional feature space.

FIG. 6B is an illustration of a Gaussian mixture model configured toclassify the set of samples into a set of clusters of probabilitydistributions within the two-dimensional feature space.

FIG. 6C is another illustration of a Gaussian mixture model configuredto classify the set of samples into a set of clusters of probabilitydistributions within the two-dimensional feature space.

FIG. 7 is an illustration of a selection of a feature subset for aclassifier from a feature set of features within a feature space basedon a correlation of respective features with respective classes inaccordance with some example embodiments.

FIG. 8 is an illustration of a classification of tumors of differentclasses based on a feature subset in accordance with some exampleembodiments.

FIG. 9 is an illustration of a Kaplan Meier survivability plot based onimage analysis in accordance with some example embodiments.

FIG. 10 is an illustration of a selection of a clinical feature subsetfor a Cox proportional hazards model from a clinical feature set ofclinical features within a feature space based on a correlation ofrespective features with respective classes in accordance with someexample embodiments.

FIG. 11 is an illustration of a Kaplan Meier survivability plot based onimage analysis and a Cox proportional hazards model in accordance withsome example embodiments.

FIG. 12 is an illustration of a result set of a classification of atumor training data set and a tumor test data set based on an imageanalysis and a Cox proportional hazards model in accordance with someexample embodiments.

FIG. 13 is a flow diagram of a first example method, in accordance withsome example embodiments.

FIG. 14 is a flow diagram of a second example method, in accordance withsome example embodiments.

FIG. 15 is a component block diagram of an example apparatus, inaccordance with some example embodiments.

FIG. 16 is a component block diagram of another example apparatus, inaccordance with some example embodiments.

FIG. 17 is an illustration of an example computer-readable medium, inaccordance with some example embodiments.

FIG. 18 is an illustration of an example apparatus in which some exampleembodiments may be implemented.

DETAILED DESCRIPTION A. Introduction

The following introduction is intended to provide an overview of somemachine learning features that relate to some example embodiments.

FIG. 1 is an example of a convolutional neural network (CNN) 110 that istrained to process an n-dimensional input to detect a number offeatures.

In the example of FIG. 1 , the convolutional neural network 110processes images 102 as a two-dimensional array of pixels 108 in one ormore colors, but some such convolutional neural networks 110 may processother forms of data, such as sound, text, or a signal from a sensor.

In the example of FIG. 1 , a training data set 100 is provided as a setof images 102 that are each associated with a class 106 from a class set104. For example, the training data set 100 may include a first image102-1 of a vehicle that is associated with a first class 106-1 forimages of vehicles; a second image 102-2 of a house that is associatedwith a second class 106-2 for images of houses; and a third image 102-3of a cat that is associated with a third class 106-3 for images of cats.The associations of the images 102 with the corresponding classes 106are sometimes known as labels of the training data set 100.

As further shown in FIG. 1 , each image 102 may be processed by aconvolutional neural network 110 that is organized as a series ofconvolutional layers 112, each having a set of neurons 114 and one ormore convolutional filters 116. In the first convolutional layer 112-1,each neuron 114 may apply a first convolutional filter 116-1 to a regionof the image, and may output an activation that indicates whether thepixels in the region corresponds to the first convolutional filter116-1. The collection of activations produced by the neurons 114 of thefirst convolutional layer 112-1, known as a feature map 118-1, may bereceived as input by a second convolutional layer 112-2 in the sequenceof convolutional layers 112 of the convolutional neural network 110, andthe neurons 114 of the second convolutional layer 112-2 may apply asecond convolutional filter 116-2 to the feature map 118-1 produced bythe first convolutional layer 112-1 to produce a second feature map118-2. Similarly, the second feature map 118-2 may be received as inputby a third convolutional layer 112-3, and the neurons 114 of the thirdconvolutional layer 112-3 may apply a third convolutional filter 116-3to the feature map 118-2 produced by the second convolutional layer112-2 to produce a third feature map 118-3. Such machine learning modelsthat include a significant plurality of layers or more complexarchitectures of layers are sometimes referred to as deep learningmodels.

As further shown in FIG. 1 , the third feature map 118-3 produced by thethird and final convolutional layer 112-3 may be received by aclassification layer 120, such as a “dense” or fully-connected layer,which may perform a classification of the third feature map 118-3 todetermine a classification of the content of the image 102-1. Forexample, each neuron 114 of the classification layer 120 may apply aweight to each activation of the third feature map 118-3. Each neuron114 outputs an activation that is a sum of the products of eachactivation of the third feature map and the weight connecting the neuron114 with the activation. As a result, each neuron 114 outputs anactivation that indicates the degree to which the activations includedin the third feature map 118-3 match the corresponding weights of theneuron 114. Further, the weights of each neuron 114 are selected basedon the activations of third feature maps 118-3 that are produced by theimages 102 of one class 106 of the class set 104. That is, each neuron114 outputs an activation based on a similarity of the third feature map118-3 for a currently processed image 102-1 to the third feature maps118-3 that are produced by the convolutional neural network 110 for theimages 102 of one class 106 of the class set 104. A comparison of theoutput of the neurons 114 of the classification layer 120 may permit theconvolutional neural network 110 to perform a classification 122 bychoosing the class 106-4 with the highest probability of correspondingto the third feature map 118-3. In this manner, the convolutional neuralnetwork 110 may perform a classification 122 of the image 102-1 as theclass 106 of images 102 that most closely resemble the image 102.

As further shown in FIG. 1 , a training process may be applied to trainthe convolutional neural network 110 to recognize a class set 104 thatis represented by a particular set of images 102 of a training data set100. During the training process, each image 102 of the training dataset 100 may be processed by the convolutional neural network 110,resulting in a classification 122 of the image 102. If theclassification 122 is incorrect, the convolutional neural network 110may be updated by adjusting the weights of the neurons 114 of theclassification layer 120 and the filters 116 of the convolutional layers112 such that the classification 122 of the convolutional neural network110 is closer to the correct classification 122 for the image 102 beingprocessed. Repeatedly training the convolutional neural network 110 onthe training data set 100, while incrementally adjusting theconvolutional neural network 110 to produce a correct classification 122for each image 102, may result in convergence of the convolutionalneural network 110, wherein the convolutional neural network 110correctly classifies the images 102 of the training data set 100 withinan acceptable range of error. Examples of convolutional neural networkarchitectures include ResNet and Inception.

As discussed with respect to FIG. 1 , machine learning models such asconvolutional neural networks 110 may be capable of classifying inputs,such as images 102, based on an arrangement of features with respect toone another, such as a characteristic number, orientation, andpositioning of recognizable features. For example, a convolutionalneural network 110 may classify an image 102 by producing a firstfeature map 118-1 indicating the detection of certain geometric shapes,such as curves and lines, that occur in various locations within theimage 102; a second feature map 118-2 indicating that the geometricshapes are arranged to produce certain higher-level features, such as aset of curves arranged as a circle or a set of lines arranged as arectangle; and a third feature map 118-3 indicating that thehigher-level features are arranged to produce even higher-levelfeatures, such as a set of circles arranged as a wheel and a set ofrectangles arranged as a door frame. A neuron 114 of the classificationlayer 120 of the convolutional neural network 110 may determine that thefeatures of the third feature map 118-3 (such as two wheels positionedbetween two door frames) are arranged in such a manner as to depict theside of a vehicle such as a car. Similar classification 122 may occur byother neurons 114 of the classification layer 120 to classify images 102as belonging to other classes 106 of the class set 104, such as anarrangement of two eyes, two triangular ears, and a nose that depicts acat, or an arrangement of windows, a door frame, and a roof that depictsa house. In this manner, machine learning models such as convolutionalneural networks may classify inputs (such as images 102) based upon anarrangement of features that correspond to similar arrangements offeatures as depicted in the inputs (such as images 102) of a trainingdata set 100. Additional details about convolutional neural networks andother machine learning models, including support vector machines, may befound in U.S. Patent Application 62/959,931, which is incorporated byreference as if fully rewritten herein.

B. Distribution-Based Classification

In some machine learning scenarios, a classification of an input (suchas an image) may occur based on an arrangement of recognizable featureswith respect to one another, such as a number, orientation, and/orpositioning of recognizable patterns of pixels as detected in a featuremap 118 of a convolutional neural network 110. However, in some otherscenarios, the classification may not be based on an arrangement offeatures with respect to one another, but instead based on adistribution of features in the input, such as whether a densityvariance of a feature over the area of the input correspond to arecognizable density variation that is characteristic of a class 106.That is, the classes 106 of a class set 104 might not be recognizable asa set of lower-level features corresponds to a recognized arrangement(e.g., number, orientation, and/or positioning) of higher-level featureswith respect to one another that corresponds to a class 106. Instead,each class 106 may be recognizable as a correspondence of thedistribution of the activation of a feature with some properties of theinput. Such distribution may not reflect any particular number,orientation, and/or positioning of the activations of features of theinput, but, rather, may indicate whether the distribution of theactivation of the feature corresponds to the distribution of theactivation of the feature for respective classes 106. In such scenarios,the inputs of each class 106 (such as a training data set) may beassociated with a characteristic distribution of the activation of thefeature, and a classification of an input may be based upon whether thedistribution of the activation of the feature of the input correspondsto the distribution of the activation of the feature among the inputs ofeach class 106. Such distribution-based classification may also arise ina variety of scenarios.

FIGS. 2A and 2B together show an example of several types of imageanalysis that may be used to identify area types and distributions oflymphocytes in an image of a tumor in some example embodiments.

FIG. 2A is an illustration of an example image analysis to identify areatypes and distributions of lymphocytes in an image of a tumor inaccordance with some example embodiments. As shown in FIG. 2A, a dataset may include an image 102 of tissue of an individual with a type oftumor, as well as stroma that includes connective tissue and support forthe tumor. Classification of the features of the image 102 may enable adetermination of areas, for example, portions of the image 102 that haveareas with similar features. A further identification of areas of thefeature map 118 that include a certain feature of a filter 116 mayenable a determination 200 of area types of the respective areas, suchas a first area the image 102 that depicts a tumor, and a second area ofthe image 102 that depicts stroma. Each filter 116 of the convolutionalneural network 110 may therefore be regarded as a mask that indicatesthe areas of the image 102 of a particular area type, such as thepresence, size, shape, and extent of a tumor, or of stroma that isadjacent to a tumor.

Further, the image 102 may show the presence of lymphocytes, which maybe distributed with regard to the tumor, stroma, and other tissue.Further analysis of the image may enable a determination 202 oflymphocyte clusters 204 as contiguous areas and/or as areas in which aconcentration of lymphocytes is high (for example, compared with theconcentration of lymphocytes in other parts of the image, or with aconcentration threshold), for example, by counting the number oflymphocytes present within a particular area of the image 102. Thus, thelowest convolutional layers 112 and filters 116 of a convolutionalneural network 110 may be capable of identifying features that areindicative of tumor, stroma, and lymphocytes.

FIG. 2B is an illustration of an example image analyses to classify thedistribution of lymphocytes in an image of a tumor in accordance withsome example embodiments. In FIG. 2B, a first image analysis 206 may beperformed by first partitioning the image 102 into a set of areas, andclassifying each area of the image 102 as tumor, tumor-adjacent, stroma,stroma-adjacent, or elsewhere. As a result, each area that includes alymphocyte cluster 204 may further characterize the lymphocyte cluster204 based on the area type, for example, a first lymphocyte cluster204-3 that occurs within a tumor and a second lymphocyte cluster 204-4that occurs within stroma.

A second image analysis 208 may be performed to further compare thelocations of lymphocyte clusters 204 with the locations of differentarea types to further characterize the lymphocyte clusters 204. Forexample, the first lymphocyte cluster 204-3 may be identified asoccurring within a central part of a tumor area, and/or within a firstthreshold distance of a location identified as a center of mass of thetumor, and may therefore be characterized as a tumor-infiltratinglymphocyte (TIL) cluster. Similarly, the second lymphocyte cluster 204-4may be identified as occurring within a central part of a stroma area,and therefore representing a stroma-infiltrating lymphocyte cluster.However, a third cluster 204-5 may be identified as occurring within aperipheral part of a tumor area, and/or within a second thresholddistance of the tumor (the second threshold distance being larger thanthe first threshold distance), and may therefore be characterized as atumor-adjacent lymphocyte cluster. Alternatively or additionally, thethird cluster 204-5 may be identified as occurring within a peripheralpart of a stroma area, and/or within a second threshold distance of thestroma (the second threshold distance being larger than the firstthreshold distance), and may therefore be characterized as astroma-adjacent lymphocyte cluster. Some example embodiments mayclassify the area as tumor, stroma, or lymphocytes; with two labels,such as tumor and lymphocytes, stroma and lymphocytes, or tumor andstroma; and/or with three labels, such as tumor, stroma, andlymphocytes. Some example embodiments may then be configured to identifyclusters 204 of lymphocytes that appear in each area of the image 102,and to tabulate the areas to determine the distribution. In this manner,the image analysis of the image 102, including the feature maps 118provided by different filters 116 of the convolutional neural network110, may be used to identify and characterize the distribution and/orconcentration of lymphocyte sin the image of the tumor in some exampleembodiments.

FIG. 3 is an illustration of a mask set 300 of masks 302 of lung tissuesamples including a lymphocyte distribution of lymphocytes by an examplemachine learning model in accordance with some example embodiments. Asshown in FIG. 3 , masks 302 of the image 102 may be prepared, each mask302 indicating the area of the image 102 that correspond to one or morearea types. For example, a first mask 302-1 may indicate areas of theimage 102 that are identified as tumor areas. A second mask 302-2 mayindicate areas of the image 102 that are identified as stroma areas. Athird mask 302-3 may indicate areas of the image 102 that are identifiedas lymphocyte areas. Still further masks may be characterized based onthe distribution of the features in the feature maps 118. For example, afourth mask 302-4 may indicate areas of the image 102 that areidentified as tumor-infiltrating lymphocyte areas. A fifth mask 302-5may indicate areas of the image 102 that are identified astumor-adjacent lymphocyte areas. A sixth mask 302-6 may indicate areasof the image 102 that are identified as stroma-infiltrating lymphocytesareas. A seventh mask 302-7 may indicate areas of the image 102 that areidentified as stroma-adjacent lymphocytes areas. An eighth mask 302-8may indicate areas of the image 102 that are identified as overlappingstroma areas and tumor areas. A ninth mask 302-9 may indicate areas ofthe image 102 that are identified as tumor-adjacent stroma areas.

In some example embodiments, the image 102 may also be processed todetermine a variety of measurements 304 of the respective area types ofthe image 102. For example, a concentration of each area type, as apercentage of the image 102, may be calculated (e.g., the number ofpixels 108 corresponding to each area as compared with the total numberof pixels of the image 102, optionally taking into account an apparentconcentration of the features, such as a density or count of lymphocytesin respective areas of the image 102). In this manner, the imageanalysis of the image 102 based on the distribution analysis shown inFIGS. 2A and 2B may be aggregated as a mask set 300 of masks 302 and/orquantities in some example embodiments.

FIG. 4 is an illustration of an example machine learning model thatclassifies tumors in accordance with some example embodiments. Theexample machine learning model of FIG. 4 includes a first convolutionalneural network 112-1 configured to perform an area classification 402 ofrespective areas 400 of an image 102 of a tumor according to differentclasses, such as tumor areas, stroma areas, lymphocyte areas,tumor-infiltrating lymphocyte areas, etc. Based on the areaclassification 402, a mask set 300 of masks 302 may be generated, forexample, a first mask 302-1 indicating areas 400 of the image 102 thatare tumor areas, a second mask 302-2 indicating areas 400 of the image102 that are stroma areas, and a third mask 302-3 indicating areas 400of the image 102 that are lymphocyte areas. The example system of FIG. 4includes a second convolutional neural network 112-2 configured todetermine a density or concentration of various features, such as alymphocyte density range estimation 404 indicating a count or percentageof lymphocytes in respective areas 400 of the image 102. Based on thelymphocyte density range estimation 404, a lymphocyte density map 406may be generated that indicates the areas 400 of the image 102 having ahigh density of lymphocytes, such as lymphocyte clusters. Based on themasks 302 of the mask set 300, area classification 402, and/or thelymphocyte density map 406 of the lymphocyte density range estimation404, an image evaluator 408 may identify aggregated areas of the image102, such as a tumor area 410-1, a stroma area 410-2, and a lymphocytearea 410-3; one or more measurements of the areas, such as a tumormeasurement 304-1, a stroma measurement 304-2, and a lymphocytemeasurement 304-3; and/or one or more areas indicating a distribution ofthe features, such as a tumor-infiltrating lymphocyte area, atumor-adjacent lymphocyte area 412-1, a tumor and stroma area, and atumor-adjacent stroma area 412-2.

To recap, in some such example embodiments, one or more convolutionalneural networks may be trained to determine a lymphocyte distribution oflymphocytes in an area of an image, for example, to classify an area ofthe image as one or more area types selected from an area type setincluding a tumor area, a lymphocyte, area, or a stroma area. In someexample embodiments, the convolutional neural network may determine thelymphocyte distribution of lymphocytes in the tumor includes, forrespective lymphocyte areas of the image; determine a distance of thelymphocyte area to one or both of a tumor area or a stroma area; andbased on the distance, characterize the lymphocyte area as one of, atumor-infiltrating lymphocyte area, a tumor-adjacent lymphocyte area, astroma-infiltrating lymphocyte area, or a stroma-adjacent lymphocytearea. In some example embodiments, the convolutional neural network maydetermine the lymphocyte distribution of lymphocytes in the tumor by,for respective stroma areas of the image, determining a distance of thestroma area to a tumor area, and based on the distance, characterizingthe stroma area as one of, a tumor-infiltrating stroma area, or atumor-adjacent stroma area, and the classifier further classifies thetumor based on the characterizing of the stroma area. The classifier maythus further classify the tumor based on the characterizing of thelymphocyte area. Many such convolutional neural networks may perform avariety of analyses of the image that may inform a determination of aclinical value (such as a prognosis) for an individual in some exampleembodiments.

C. Learning Parameter Determination

FIG. 5 is an illustration of a characterization 500 of a set of imagesof pancreatic adenocarcinoma tissue samples in accordance with someexample embodiments. In the charts of FIG. 5 , a set of tumors ischaracterized by the system shown in FIG. 4 to determine a distributionof the detected features of tumor, such as tumor areas 502-1, stromaareas 502-2, lymphocyte areas 502-3, tumor-invasive lymphocyte areas502-4, tumor-adjacent lymphocyte areas 502-5, stroma- and tumor-invasivelymphocyte areas 502-6, stroma-adjacent lymphocyte areas 502-7, tumorand stroma areas 502-8, and tumor-adjacent stroma areas 502-9. Eachfeature may be evaluated as to both a density or concentration (verticalaxis) and a percentage (horizontal axis) of each feature in the images102 of the tumors. The set of images of tumors may be further dividedinto a subset of training images, which may be used to train a machinelearning classifier such as the neural networks 112-1, 112-2 todetermine the features, and a subset of testing images, which may beused to evaluate the effectiveness of the machine learning classifiersin determining the features in previously unseen images of tumors. Inthis manner, the machine learning classifier may be validated todetermine the consistency of the underlying logic when applied to newdata. For example, the chart in FIG. 5 was developed based on diagnostichematoxylin and eosin stain (H&E-stain) of pathology images ofpancreatic adenocarcinoma patients who underwent chemotherapy.

It may be further desirable to characterize the tumors as one of severalclasses, such as low-risk tumors and high-risk tumors, on the basis offactors that are characteristic of tumors of each class. For example,tumors of respective classes may also be associated with differentfeatures, such as concentration and/or percentage of a particular typeof tumor area (e.g., tumor-invasive lymphocytes), and differences insuch characteristic features may enable the tumors of one class to bedistinguished from tumors of another class. Further, different tumorclasses may be associated with different clinical properties, such asresponsiveness to various treatment options and prognosis such assurvivability. In order to determine such clinical properties for aparticular tumor in an individual, it may be desirable to determine thetumor class of the tumor in order to guide the selection of a diagnosisand/or treatment regimen for the individual.

However, in many diagnostic scenarios, it may be difficult to associatethe features that are characteristic of the different classes, such asdifferent tumor classes. As a first such example, the features of tumorsin one class may vary from the features of tumors in another classwithin a probabilistic range, and the probabilistic ranges may overlapby a significant amount. For example, the characteristic density andpercentages of tumor-invasive lymphocytes for a high-risk tumor classand a low-risk tumor may each fit a bell curve of probability within thetumor class, and the means of the bell curves being only marginallyoffset, such that the probabilistic distributions may overlap. It maytherefore be difficult to determine whether a tumor exhibiting thefeature within the overlapping areas is of the high-risk class or thelow-risk class. As a second such example, different features of tumorsmay covary; for example, high-risk tumors may be distinguished fromlow-risk tumors based on the combined probabilities of distinguishingdistributions of tumor-invasive lymphocyte areas and tumor-adjacentlymphocyte areas. However, different features may also innately covaryin ways that are not diagnostic. For example, tumors that exhibit a highdensity of stroma- and tumor-invasive lymphocyte areas also necessarilyexhibit a high density of tumor-invasive lymphocyte areas in general. Asa result, adding the class-based probability of a tumor belonging to aclass on the basis of stroma- and tumor-invasive lymphocyte areas andthe class-based probability of a tumor belonging to a class on the basisof tumor-invasive lymphocyte areas may overweigh the likelihood of atumor being in the class, due to failing to account for the innatecovariance of the features. Due to these complex features of the data,it may be difficult to determine the distinguishing features for eachclass of tumors, particularly in high-dimensionality feature sets wheremany features may be available.

In order to classify a data set that exhibits such overlapping classesof data, a variety of machine learning models may be used. Respectivemachine learning models may provide different capabilities ofclassifying the overlapping data sets, for example, on the basis ofdistinctiveness, tolerance for false positives, tolerance for falsenegatives, scalability to larger numbers of features, and avoidance ofproperties such as overfitting and underfitting.

FIGS. 6A-6C together show an example of a Gaussian mixture model thatmay be developed to classify overlapping classes of data, such asclassifying tumors into low-risk tumors and high-risk tumors on thebasis of two features, which may be used in some example embodiments.

FIG. 6A is an illustration of a set of samples arranged in atwo-dimensional feature space 606. In FIG. 6A, the feature space 606involve samples 600-1 of a first class 602-1 (represented as circles)and samples 600-2 of a second class 602-2 (represented as crosses). Eachsample 600 may be evaluated and quantified as to a first feature 604-1and a second feature 604-2, which may enable each sample to bepositioned within the two-dimensional feature space 606, wherein thevertical axis represents the first feature 604-1 and the horizontal axisrepresents the second feature 604-2. Within the two-dimensional featurespace 606, the samples 600 of each class 602 may be apparentlyclustered, but the clusters may also overlap, such that samples withinthe overlapping part may belong to either class 602. For a particularsample 600, it may be desirable to determine a probability that thesample 600 belongs in each class 602 based upon the features 604 of thesample 600, particularly in the overlapping area that is associated withsamples 600 of multiple classes 602. While the clustering may beapparent in the simple illustration of FIG. 6A, such clustering may bemore difficult to determine, for example, in feature spaces 606 withhigher dimensionality, in data sets featuring classes 602 with a greaterdegree of overlap, and/or in data sets in which two or more features 604covary for which determining the diagnostic or innate covariance of thefeatures 604.

A variety of machine learning models may be used to classify overlappingdata sets, such as shown in FIG. 6A. Some such models include, forexample, Bayesian (including naïve Bayesian) classifiers; Gaussianclassifiers; probabilistic classifiers; principal component analysis(PCA) classifiers; linear discriminant analysis (LDA) classifiers;quadratic discriminant analysis (QDA) classifiers; single-layer ormultiplayer perceptron networks; convolutional neural networks;recurrent neural networks; nearest-neighbor classifiers; linear SVMclassifiers; radial-basis-function kernel (RBF) SVM classifiers;Gaussian process classifiers; decision tree classifiers, includingrandom forest classifiers; and/or restricted or unrestricted Boltzmannmachines, among others

FIG. 6B is an illustration of a Gaussian mixture model configured toclassify the set of samples 600 shown in FIG. 6A into a set of clustersof probability distributions within the two-dimensional feature space606, and which may be used to distinguish different classes of tumors insome example embodiments. In FIG. 6B, a first Gaussian probabilitydistribution 608-1 may be identified for the samples 600-1 of the firstclass 602-1, and a second Gaussian probability distribution 608-2 may beidentified for the samples 600-2 of the second class 602-2. The Gaussianprobability distributions 608 for each class 602 may be fit to thesamples 600 of each class 602, for example, based on the mean andvariance of the samples 600 for each feature 604. The selection of theGaussian probability distributions 608 may also take into considerationother factors such as avoiding false negatives (e.g., samples 600 of theclass 602 being incorrectly excluded from the class 602) and/or avoidingfalse positives (e.g., samples 600 of a different class 602 beingincorrectly included in the class 602). Further, the Gaussianprobability distributions 608 may be selected to model covariance, forexample, by associating the distribution of the Gaussian probabilitydistribution 608 for the first feature 604-1 and the distribution of theGaussian probability distribution 608 for the second feature 604-2. Forexample, a similar deviation and/or of the Gaussian probabilitydistribution 608 may be selected for the first feature 604-1 and thesecond feature 604-2, or may be independently selected for each feature604. For a particular sample 600 (such as an image of a tumor of anunknown class), the features 604 of the sample 600 may be evaluated toposition the sample 600 within the feature space 606, and the relativeprobabilities within the Gaussian probability distributions 608 of therespective classes 602 may be compared to determine a likely class 602of the tumor.

As further shown in FIG. 6B, the fitness of the selected Gaussianmixture model may also be evaluated, for example, as an estimate of thediagnostic properties of the Gaussian mixture model. For example, asilhouette score 610 may be determined for each Gaussian probabilitydistribution 608, where the silhouette score indicates a silhouettecoefficient 612 (e.g., the number of samples 600 of the class 602 thatare within a selected distance from the mean or center of mass of theGaussian probability distribution 608). The distinguishing properties ofthe Gaussian mixture model may be improved by selecting Gaussianprobability distributions 608 with similar silhouette scores 610. Asshown in FIG. 6B, the silhouette scores of the Gaussian probabilitydistributions 608 are dissimilar, for example, because the firstGaussian probability distribution 608-1 for the first class 602-1represents a larger number of samples 600-1 than the second Gaussianprobability distribution 608-2 for the samples 600-2 of the second class602-2 (that is, a taller silhouette for the first Gaussian probabilitydistribution 608-1 than for the second Gaussian probability distribution608-2), and also because the distances of the samples 600-1 of the firstclass 602-1 are more widely distributed in the feature space 606 thanthe samples 600-2 of the second class 602-2, leading to a larger rangeof silhouette coefficients 612 (that is, a longer silhouette for thefirst Gaussian probability distribution 608-1 than for the secondGaussian probability distribution 608-2). As a result, the Gaussianmixture model of FIG. 6B may be improved by selecting a differentmixture of Gaussian probability distributions 608.

FIG. 6C is another illustration of a Gaussian mixture model configuredto classify the set of samples into a set of clusters of probabilitydistributions within the two-dimensional feature space, and which may beused to distinguish different classes of tumors in some exampleembodiments. In FIG. 6C, the Gaussian probability distributions for thefirst class 602 are instead identified as a first Gaussian probabilitydistribution 608-4 for a first cluster of samples 600-1 of the firstclass 602-1 and a second Gaussian probability distribution 608-5 for asecond cluster of samples 600-1 of the first class 602-1. Further, foreach of the Gaussian probability distributions 608, a mixing parametermay be identified that indicates the proportion of samples 600 of theclass 602 that are represented by the Gaussian probability distribution608. For example, the first Gaussian probability distribution 608-4 forthe first class 602-1 may fit a smaller number of samples 600-1 of thefirst class 602-1 than the second Gaussian probability distribution608-5 for the first class 602-1, and may therefore have a smaller firstmixing parameter 614-1 than a second mixing parameter 614-2 for thesecond Gaussian probability distribution 608-5. When a sample 600 of anunknown class 602 is positioned within the feature space 606, theprobability of the sample 600 being classified into each class 602 maybe determined as the sum of the products of the probabilitydistributions for the position of the sample 600 by each Gaussianprobability distribution 608 and the mixing parameter 614 for theGaussian probability distribution 608. Further, the classifyingcapability of the Gaussian mixture model may be evaluated based on thesilhouette scores 610 of the respective Gaussian probabilitydistributions 608; for example, the similarities of both sample size andsilhouette coefficients 612 for each Gaussian probability distribution608, may indicate a more reliable and predictive classifier than theGaussian mixture model of FIG. 6B.

Alternatively or in addition to the silhouette scores shown in FIGS. 6Band 6C, other measures may be used to determine the classifyingcapabilities of a Gaussian mixture model. As one example, for tumors ofdifferent tumor classes (such as a low-risk class and a high-risk class)that are respectively associated with survivability, a concordance index(“C-index”) may be developed that indicates a degree of consistencybetween a predicted survival time of individuals with tumors of a tumorclass and the actual survival times of individuals with tumors of thetumor class. Concordance indices may be determined on the basis of eachfeature of the tumor class to determine the degree to which the featurecorresponds to the predicted survival rate of the individuals withtumors in the tumor class, where a high concordance index indicates ahighly predictive feature of the Gaussian mixture model and a lowconcordance index indicates a poorly predictive feature of the Gaussianmixture model. Because the concordance index of each feature dependsupon the selected Gaussian mixture model, it may be desirable to limitthe number of features to those that exhibit a high concordance index,alternatively or additionally to the silhouette scores of the respectiveGaussian probability distributions for each class. Selecting suchfeatures may reduce the dimensionality of the feature space 606 of thedata set to a smaller set of features that are more highlydistinguishing for the respective classes 602, which may yield a moreprecise, accurate, and/or efficient classification process.

FIG. 7 is an illustration of a selection process 700 for selecting aclinical feature subset for a classifier from a clinical feature set ofclinical features within a feature space based on a correlation ofrespective clinical features with respective classes in accordance withsome example embodiments. A Gaussian mixture model is developed for aset of nine clinical features, such as the nine clinical features shownin FIG. 5 . A set of silhouette scores and concordance indices may bedetermined for each clinical feature. Among a set of available clinicalfeatures 704, in a first selection step 702-1, a first clinical feature706-1 may be selected that provides with a highest silhouette scoreand/or concordance index among the available clinical features 704, sucha concentration (specifically, percentage) of tumor-adjacent lymphocyteareas. Among the remaining clinical features (that is, all of theclinical features except the first selected clinical feature), in asecond selection step 702-2, a second Gaussian mixture model may bedeveloped, and a second clinical feature 706-2 may be selected thatprovides with a highest silhouette score and/or concordance index amongthe remaining clinical features, such a concentration (specifically,percentage) of stroma and tumor areas. Similar selection steps 702-3,702-4 may be performed to select a third clinical feature 706-3 (such asconcentration of tumor-invasive stroma areas) and a fourth clinicalfeature 706-4 (such as concentration of lymphocytes), each of whichprovides an improved concordance score as compared with the previouslyselected clinical features, indicating a supplemental classificationcapability of the selected clinical feature as compared with the otherremaining clinical features. The selection process may continue until afifth selection step 702-5, in which the selected clinical feature isdetermined not to improve upon the concordance indices of the previouslyselected clinical features, and no further clinical features may beselected for the clinical feature subset.

To recap, in some example embodiments, a classifier for a tumor mayinclude Gaussian mixture model configured to determine, for respectiveclasses, a probability distribution of features for tumors in the classwithin a feature space, which may be selected from a feature setincluding a measurement of tumor areas of the image, a measurement ofstroma areas of the image, a measurement of lymphocyte areas of theimage, a measurement of tumor-infiltrating lymphocyte areas of theimage, a measurement of tumor-adjacent lymphocyte areas of the image, ameasurement of stroma-infiltrating lymphocyte areas of the image, ameasurement of stroma-adjacent lymphocyte areas of the image, ameasurement of tumor-infiltrating stroma areas of the image, and ameasurement of tumor-adjacent stroma areas of the image. In some exampleembodiments, a feature subset for the Gaussian mixture model may beselected based on a correlation of the respective classes withrespective features of the subset, wherein the correlation may be basedon one or both of, a silhouette score of the feature space or aconcordance index. In some example embodiments, the feature subset mayconsist essentially of the measurement of lymphocyte areas of the image,the measurement of tumor-infiltrating lymphocyte areas of the image, themeasurement of tumor-adjacent lymphocyte areas of the image, and themeasurement of tumor-infiltrating stroma areas of the image.

D. Image-Based Tumor Evaluation

In some example embodiments, a clinical value (such as a prognosis) foran individual based on a tumor shown in an image may be determined bydetermining a lymphocyte distribution of lymphocytes in the tumor basedon the image; applying a classifier to the lymphocyte distribution toclassify the tumor, the classifier having been trained to classifytumors into a class selected from at least two classes respectivelyassociated with lymphocyte distributions; and determining the clinicalvalue (such as the prognosis) for the individual based on prognoses ofindividuals with tumors in the class into which the classifierclassified the tumor. The classifier may be invoked to determine thelymphocyte distribution of lymphocytes in respective areas of the imageof the tumor.

FIG. 8 is an illustration of a classification of tumors of differentclasses based on a feature subset in accordance with some exampleembodiments. FIG. 8 presents a comparison 800 of the feature subset ofselected features 806 with the images 804 of tumors of a low-risk tumorclass 802-1 and a high-risk tumor class 802-2, that is, in thepercentages of areas in each image 804 of each class 802 correspondingto each of the features 806 of the feature subset. The high-risk classof tumors may be associated with a first survival probability, and thelow-risk class of tumors may be associated with a second survivalprobability that is longer than the first survival probability. Thepercentages of the respective features 806 of the images 804 of thetumor classes 802 may be compared, for example, to determine the degreeto which the features 806 are diagnostic of the respective tumor classes802. For example, the images 804 of the high-risk tumor class 802-2 maydemonstrate a smaller and more consistent range of values for the firstfeature 806-1 and the third feature 806-2 than for the images 804 of thelow-risk tumor class 802-1. Also, the values for the third feature 806-3may be typically higher in images of tumors of the low-risk tumor class802-1 than in images of tumors of the high-risk tumor class 802-2. Thesemeasurements, which may be determined based on selected features 706 ofthe feature subsets of the tumor classes 802 based on the selectionprocess 700 of FIG. 7 , may present clinically significant findings inthe pathology of tumors of different tumor classes 802, and may be usedby both clinicians and automated processes (such as diagnostic and/orprognostic machine learning processes) to classify tumors into differenttumor classes 802.

FIG. 9 is an illustration of a Kaplan Meier survivability plot based onimage analysis in accordance with some example embodiments. In FIG. 9 ,a first Kaplan Meier survivability plot 900-1 and a second Kaplan Meiersurvivability plot 900-2 (e.g., percentages of surviving populations ofindividuals as measured by days after diagnosis) are generated,respectively, for a training set and test set based on a population ofindividuals having tumors of the low-risk tumor class 802-1 and thehigh-risk tumor class 802-2 of FIG. 8 . Further, the set of tumors forwhich images and data are available is separated into a training set anda test set. The machine learning models, including the convolutionalneural networks and/or the Gaussian mixture models, are trained on theimages of the training set to a convergence point in which the machinelearning models produce output that is within an accuracy range of theexpected output. The machine learning models are then tested using thetest set to determine whether the machine learning models produce outputfor new data that is consistent with the expected output. Suchvalidation may include cross-validation processes in which the set oftumors is first partitioned into a number of subsets, and repeatedtraining and testing are performed using a selection from the subsetsfor the training set and the remaining subsets for the test set.

As shown in FIG. 9 , the image-based tumor evaluation techniquepresented herein performed classification on the training data set witha hazard ratio (HR) of 0.5117, a statistical P-value of 0.0570, and aconcordance index of 0.6667, and demonstrated performance on the testset with a hazard ratio of 0.5154, a statistical P-value of 0.3405, anda concordance index of 0.5964. Many such machine learning models may betrained to classify tumors and to determine the clinical value (such asthe prognosis and/or the survivability) for the individual in accordancewith some example embodiments.

E. Cox Proportional Hazards Model

In some example embodiments, the image-based prognosis determinationtechniques may be combined with a Cox proportional hazards model, whichmay improve the prognostic capabilities of tumor analysis. The Coxproportional hazards model is a regression model that correlatesclinical features, such as the individual's demographic features,clinical observations of the individual and the tumor, and pathologymeasurements, with different tumor classes to determine the contributionof each clinical feature to the tumor classification. For example, theregression model may determine that individuals within a particular agerange, with particular personal habits such as smoking or alcohol usage,and with a cancer staging score, such as based on the American JointCommittee on Cancer (AJCC) cancer staging system, are more likely to beclassified with tumors within a low-risk tumor class, while individualswithin another age range, having other personal habits, and with othercancer staging scores are more likely to be classified with tumorswithin a high-risk tumor class.

A Cox proportional hazards model may be developed using a training setfeaturing tumors with known clinical features. A stepwise selection maybe performed to select a subset of clinical features that significantlycontribute to classification, for example, by removing the clinicalfeatures that do not significantly improve the predictiveness of theother clinical features. The Cox proportional hazards model may also betrained on the tumors of two or more classes to determine differentproportional survivability rates for tumors of different tumor classes,such as a low-risk class of tumors having a shared set of propertiesand/or similar survivability metrics and a high-risk class of tumorshaving another shared set of properties and/or other similarsurvivability metrics.

FIG. 10 is an illustration of a selection of a feature subset for a Coxproportional hazards model from a feature set 1000 of features within afeature space based on a correlation of respective features withrespective classes in accordance with some example embodiments. In FIG.10 , for respective tumors of a set of tumors taken from individuals andpathologically evaluated, the values for the clinical feature set 1000are identified that includes a primary diagnosis of the tumor (e.g., aT-category AJCC staging score); a measurement of the tumor (e.g., anN-category AJCC staging score); an ethnicity of the individual; atreatment of the tumor; a location of the tumor; a smoking habitfrequency of the individual; a metastatic condition of the tumor; a raceof the individual; a previous cancer medical history of the individual;a smoking habit duration of the individual; a primary diagnosis of theindividual; an alcohol history of the individual; and a gender of theindividual. A first step 1002-1 of regression analysis may determine theextent to which each feature of the feature set 1000 distinguishesbetween the tumor classes (e.g., low-risk and high-risk), and thefeatures may be ordered, for example, by statistical P-values. Thefeatures having P-values within a certain range (for example, below astatistical significance threshold of 0.05) may be selected as a featuresubset, and the other features may be excluded. Additional steps 1002-2,1002-3, 1004 of regression analysis may be performed to exclude otherfeatures, and to retain other features of the feature set 1000, untilfeatures can no longer be excluded without significantly reducing theclassification accuracy of the Cox proportional hazards model. Theresulting feature set 1004, based on the correlation of the respectiveclasses with respective features of the subset, may be identified as theretained features of the Cox proportional hazards model.

As shown in FIG. 10 , a Cox proportional hazards model developed in thismanner identified a feature subset consisting of the measurement of thetumor and the metastatic condition of the tumor. For a training set, theCox proportional hazards model demonstrated a hazard ratio of 0.2182 anda statistical P-value of 0.0200, and for a test set, the Coxproportional hazards model demonstrated a hazard ratio of 0.4065 and astatistical P-value of 0.2855. Many such Cox proportional hazard modelsmay be determined to classify tumors in accordance with some exampleembodiments.

F. Combined Model

In some example embodiments, image-based classification (e.g., based ona convolutional neural network and a Gaussian mixture model) may becombined with a Cox proportional hazards model to classify the tumorbased on both image features and clinical features. That is, the atleast two classes are a low-risk tumor class and a high-risk tumorclass; determining the lymphocyte distribution may further includeapplying a convolutional neural network to the image, the convolutionalneural network configured to measure the lymphocyte distribution oflymphocytes for different area types of the image; the classifier may bea two-way Gaussian mixture model configured to determine, for respectiveclasses, a probability distribution of features for tumors in the classwithin a feature space; a Cox proportional hazards model may be appliedto clinical features of the tumor to determine a class of the tumor; anddetermining the clinical value (such as the prognosis) for theindividual may be further based on the class determined by the Coxproportional hazards model.

FIG. 11 is an illustration of a Kaplan Meier survivability plot based onimage analysis and a Cox proportional hazards model in accordance withsome example embodiments. In FIG. 11 , a first Kaplan Meiersurvivability plot 900-3 and a second Kaplan Meier survivability plot900-4 (e.g., percentages of surviving populations of individuals asmeasured by days after diagnosis) are generated, respectively, for atraining set and test set based on a population of individuals havingtumors of the low-risk tumor class 802-1 and high-risk tumor class 802-2of FIG. 8 . Further, the set of tumors for which images and data,including clinical features, are available is separated into a trainingset and a test set. The machine learning models, including theconvolutional neural networks, the Gaussian mixture models, and the Coxproportional hazards model, are trained on the images of the trainingset to a convergence point in which the machine learning models produceoutput that is within an accuracy range of the expected output. Themachine learning models are then tested using the test set to determinewhether the machine learning models produce output for new data that isconsistent with the expected output. Such validation may includecross-validation processes in which the set of tumors is firstpartitioned into a number of subsets, and repeated training and testingare performed using a selection from the subsets for the training setand the remaining subsets for the test set.

As shown in FIG. 11 , the image-based tumor evaluation techniquepresented herein performed classification on the training data set witha hazard ratio (HR) of 0.2545, a statistical P-value of 0.0065, and aconcordance index of 0.7141, and demonstrated performance on the testset with a hazard ratio of 0.3742, a statistical P-value of 0.0696, anda concordance index of 0.6120.

FIG. 12 is an illustration 1200 of a result set of a classification of atumor training data set and a tumor test data set based on an imageanalysis and a Cox proportional hazards model in accordance with someexample embodiments. As shown in FIG. 12 , classification results forthe combined model featuring both image-based analysis and statisticalanalysis of clinical features demonstrate greater classificationaccuracy than for either model used alone. Many such machine learningmodels may be trained to classify tumors and to determine the clinicalvalue (such as the prognosis and/or survivability) for the individual inaccordance with some example embodiments.

G. Tumor Evaluation and Output

In some example embodiments, the tumor analysis models disclosed hereinmay be used to determine and output, for a user, a clinical value for anindividual based on a tumor shown in an image. The user may be, forexample, the individual with the tumor; a family member or guardian ofthe individual; or a healthcare provider, including a physician, nurse,or clinical pathologist. The clinical value and/or the output may be,for example, one or more of: a diagnosis for the individual, a prognosisfor the individual, a survivability of the individual, a classificationof the tumor, a diagnostic and/or treatment recommendation for theindividual, or the like.

Some example embodiments may use the determination of the tumor analysismodel to display a visualization of the clinical value (such as theprognosis) for the individual. For example, a terminal may accept animage of a tumor of an individual, and, optionally, a set of clinicalfeatures, such as the individual's demographic features, clinicalobservations of the individual and the tumor, and pathologymeasurements. The terminal may apply the tumor analysis model (e.g.,processing the image by a convolutional neural network and a Gaussianmixture model, and, optionally, processing the clinical features by aCox proportional hazards model) to determine a class of the tumor, suchas a low-risk tumor class and a high-risk tumor class, and a prognosisthat is associated with individuals with tumors of the tumor class. Theclinical value may be determined, for example, as a survivability, suchas projected survival durations and probabilities, optionally includinga confidence or accuracy of each probability. In some exampleembodiments, the clinical value may be presented as a visualization,such as a Kaplan Meier survivability projection of the tumor. In someexample embodiments, the visualization may include additionalinformation about the tumor, such as one or more of the masks 302 thatindicate the area types of the areas of the image 102; measurements 304of the image 102, such as a concentration for each area type (e.g., aconcentration of lymphocytes in one re more areas as determined bybinning), and/or a percentage area of the area type as compared with theentire image 102. In some example embodiments, the visualization mayinclude additional information about the individual, such as theindividual's clinical features, and may indicate how respective clinicalfeatures contribute to the determination of the clinical value (such asthe prognosis) for the individual.

Some example embodiments may use the determination of the tumor analysismodel to determine, and to display for a user, a diagnostic test for thetumor based on the clinical value (such as the prognosis) for theindividual. For example, based on the tumor being classified as alow-risk class by the tumor analysis model, an apparatus may recommendless aggressive testing to further characterize the tumor, such as bloodtests or imaging. Based on the tumor being classified as a high-riskclass by the tumor analysis model, an apparatus may recommend moreaggressive testing to further characterize the tumor, such as a biopsy.Some example embodiments may also display, for the user, an explanationof the basis of the determination; a set of options for further testing;and/or a recommendation of one or more options to be considered by theindividual and/or a healthcare provider.

Some example embodiments may use the determination of the tumor analysismodel to determine, and to display for a user, a treatment of theindividual based on the clinical value (such as the prognosis) for theindividual. For example, based on the tumor being classified as alow-risk class by the tumor analysis model, an apparatus may recommendless aggressive treatment of the tumor, such as less aggressivechemotherapy. Based on the tumor being classified as a high-risk classby the tumor analysis model, an apparatus may recommend more aggressivetreatment of the tumor, such as more aggressive chemotherapy and/orsurgical removal. Some example embodiments may also display, for theuser, an explanation of the basis of the determination; a set of optionsfor further testing; and/or a recommendation of one or more options tobe considered by the individual and/or a healthcare provider.

Some example embodiments may use the determination of the tumor analysismodel to determine, and to display for a user, a schedule of atherapeutic agent for treating the tumor based on the clinical value(such as the prognosis) for the individual. For example, based on thetumor being classified as a low-risk class by the tumor analysis model,an apparatus may recommend chemotherapy with a lower frequency, at alater date, and/or with a lower dosage. Based on the tumor beingclassified as a high-risk class by the tumor analysis model, anapparatus may recommend more aggressive treatment of the tumor, suchchemotherapy with a higher frequency, at an earlier date, and/or with ahigher dosage. Some example embodiments may also display, for the user,an explanation of the basis of the determination; a set of options forfurther testing; and/or a recommendation of one or more options to beconsidered by the individual and/or a healthcare provider. Many suchtypes of classification and output of clinical values for the individualand information about the tumor may be provided in some exampleembodiments.

H. Technical Effects

Some example embodiments that feature analysis using distribution-basedmachine learning classifiers may exhibit a variety of technical effects.

A first example of a technical effect that may be exhibited by someexample embodiments is a new type of input classification based upondistribution, which may be difficult to achieve through other machinelearning models. For example, as shown in FIG. 9 , an image-based tumorclassification model as disclosed herein may be capable of classifyingtumors with reasonable accuracy. As further shown in FIGS. 11 and 12 , acombined model that includes both image-based analysis (for example,based on a convolutional neural network and classification by a Gaussianmixture model) and regression-based analysis of clinical features (forexample, based on a Cox proportional hazards model) may be capable ofgreater classification accuracy than either model used alone. In somescenarios, the use of machine learning models, including a visualizationand/or explanation of the basis for such determinations of the clinicalvalue for the individual (such as an indication of the image featuresand clinical features that contribute to the determination of theprognosis), may provide an automated process for providing clinicalvalues that provide diagnostic, prognostic, and/or therapeuticinformation, and that a caregiver may utilize to choose a healthcareregimen of an individual.

A second example of a technical effect that may be exhibited by someexample embodiments is a more efficient allocation of resources basedupon such analyses. For example, classification of tumors based onautomated techniques may reduce the volume and/or dependency of clinicaland pathology resources applied to diagnose and classify tumors and todetermine clinical values (such as prognoses) of individuals. Sucheconomy of resources may also involve a faster classification processthan may be systematically achievable by classification processesperformed by individuals.

I. EXAMPLE EMBODIMENTS

FIG. 13 is a flow diagram of a first example method 1300, in accordancewith some example embodiments.

The first example method 1300 may be implemented, for example, as a setof instructions that, when executed by processing circuitry of anapparatus, cause the apparatus to perform each of the elements of thefirst example method 1300. The first example method 1300 may also beimplemented, for example, as a set of instructions that, when executedby processing circuitry of an apparatus, cause the apparatus to providea system for components, including an image evaluator, a classifier, anda tumor evaluator, that interoperate to provide a system for classifyingtumors.

The first example method 1300 includes executing 1304, by processingcircuitry of an apparatus, instructions that cause the apparatus toperform a set of elements.

For example, the execution of the instructions may cause the apparatusto determine 1306 a lymphocyte distribution of lymphocytes in the tumorbased on the image.

For example, the execution of the instructions may cause the apparatusto apply 1308 a classifier to the lymphocyte distribution to classifythe tumor, the classifier having been trained to classify tumors into aclass selected from at least two classes respectively associated withlymphocyte distributions.

For example, the execution of the instructions may cause the apparatusto determine 1310 the clinical value (such as the prognosis) for theindividual based on prognoses of individuals with tumors in the classinto which the classifier classified the tumor.

In this manner, the execution of the instructions by the processingcircuitry may cause the apparatus to perform the elements of the firstexample method 1300, and so the first example method 1300 ends.

FIG. 14 is a flow diagram of a second example method, in accordance withsome example embodiments.

The second example method 1400 may be implemented, for example, as a setof instructions that, when executed by processing circuitry of anapparatus, cause the apparatus to perform each of the elements of thesecond example method 1400. The second example method 1400 may also beimplemented, for example, as a set of instructions that, when executedby processing circuitry of an apparatus, cause the apparatus to providea system for components, including an image evaluator, a classifier, anda tumor evaluator, that interoperate to provide a system for classifyingtumors.

The second example method 1400 includes executing 1404, by processingcircuitry of an apparatus, instructions that cause the apparatus toperform a set of elements.

For example, the execution of the instructions may cause the apparatusto apply 1406 a convolutional neural network to the image to determine alymphocyte distribution of lymphocytes in the tumor, wherein theconvolutional neural network is configured to measure the lymphocytedistribution of lymphocytes for different area types of the image.

For example, the execution of the instructions may cause the apparatusto apply 1408 a classifier to the lymphocyte distribution to classifythe tumor, wherein the classifier has been trained to classify tumorsinto a class selected from a low-risk class and a high-risk class, theclasses respectively being associated with lymphocyte distributions, andthe classifier including a two-way Gaussian mixture model configured todetermine, for respective classes, a probability distribution offeatures for tumors in the class within a feature space.

For example, the execution of the instructions may cause the apparatusto apply 1410 a Cox proportional hazards model to clinical features ofthe tumor to determine a class of the tumor.

For example, the execution of the instructions may cause the apparatusto determine 1412 the clinical value (such as the prognosis) for theindividual based on the prognoses of individuals with tumors in theclass into which the classifier classified the tumor and the classdetermined by the Cox proportional hazards model.

In this manner, the execution of the instructions by the processingcircuitry may cause the apparatus to perform the elements of the secondexample method 1400, and so the second example method 1400 ends.

FIG. 15 is a component block diagram of an example apparatus, inaccordance with some example embodiments.

As shown in FIG. 15 , an example apparatus 1500 may include processingcircuitry 1502 and a memory 1504. The memory 1504 may store instructions1506 that, when executed by the processing circuitry 1502, cause theexample apparatus 1500 to determine a clinical value (such as aprognosis) for an individual based on a tumor shown in an image 102 inaccordance with some example embodiments. In some example embodiments,execution of the instructions 1506 may cause the example apparatus 1500to instantiate and/or use a set of components of a system 1508. WhileFIG. 15 illustrates one such system 1508, some example embodiments maysystem 1508 may embody any of the methods disclosed herein.

The example system 1508 of FIG. 15 includes an image evaluator 1510 thatis configured to determine a lymphocyte distribution of lymphocytes inthe image 102. For example, a class set 1516 may associate respectivelymphocyte distributions 1520-1, 1520-2 with different classes 1518-1,1518-2 of tumors, each class 1518 being associated with a prognosis1522-1, 1522-2.

The example system 1508 of FIG. 15 includes a tumor classifier 1512configured to classify tumors into a class selected from the at leasttwo classes 1518 respectively associated with the lymphocytedistributions 1520.

The example system 1508 of FIG. 15 includes a tumor evaluator 1514 thatis configured to determine a clinical value (such as a prognosis) for anindividual based on a tumor in the image 102 by invoking the imageevaluator 1510 with the image 102 to determine the lymphocytedistribution 1520-3 of lymphocytes in the tumor, invoke the tumorclassifier 1512 to classify the tumor into a class 1518 based on thelymphocyte distribution 1520-3, and output, for a user 1524, a clinicalvalue (such as a prognosis) for the individual based on the prognoses1522 of tumors in the class 1518-3 into which the tumor classifier 1512classified the tumor.

In this manner, the example apparatus 1500 and example system 1508provided thereon may classify the tumor in accordance with some exampleembodiments.

FIG. 16 is a component block diagram of another example apparatus, inaccordance with some example embodiments.

As shown in FIG. 16 , an example apparatus 1600 may include processingcircuitry 1502 and a memory 1504. The memory 1504 may store instructions1506 that, when executed by the processing circuitry 1502, cause theexample apparatus 1600 to determine a clinical value (such as aprognosis) for an individual based on a tumor shown in an image 102 inaccordance with some example embodiments. In some example embodiments,execution of the instructions 1506 may cause the example apparatus 1600to instantiate and/or use a set of components of a system 1602.

The example system 1602 of FIG. 16 includes a convolutional neuralnetwork 110, as an image evaluator, that is configured to determine alymphocyte distribution of lymphocytes in the image 102 by measuring thelymphocyte distribution of lymphocytes for different area types of theimage 102. For example, a class set 1516 may associate respectivelymphocyte distributions 1520-1, 1520-2 with different classes 1518-1,1518-2 of tumors, including a low-risk tumor class and a high-risk tumorclass, each class 1518 being associated with a prognosis 1522-1, 1522-2.

The example system 1602 of FIG. 16 includes a two-way Gaussian mixturemodel 1604, as a tumor classifier, that is configured to determine, forrespective classes 1518, a probability distribution of features fortumors in the class 1518 within a feature space 606.

The example system 1602 of FIG. 16 includes a Cox proportional hazardsmodel 1608 configured to a clinical features set 1606 of clinicalfeatures to determine a class of the tumor.

The example system 1602 of FIG. 16 includes a tumor evaluator 1514 thatis configured to determine a clinical value (such as a prognosis) for anindividual based on a tumor in the image 102 of the tumor by invokingthe convolutional neural network 110 with the image 102 to determine thelymphocyte distribution 1520-3 of lymphocytes in the tumor, invoke theGaussian mixture model 1604 to classify the tumor into a class 1518-3based on the lymphocyte distribution 1520-3, invoke the Cox proportionalhazards model 1608 with the clinical feature set 1606 for the tumor todetermine a tumor class 1518-4, and output, for a user 1524, a clinicalvalue (such as a prognosis) for the individual based on the prognoses1522 of tumors in the class 1518-5 into which the tumor classifier 1512classified the tumor based on the prognoses for the individuals withtumors in the class 1518-3 into which the Gaussian mixture model 1604classified the tumor and the tumor class 1518-4 determined by the Coxproportional hazards model 1608.

In this manner, the example apparatus 1600 and example system 1602provided thereon may classify the tumor in accordance with some exampleembodiments.

As shown in FIGS. 15 and 16 , example apparatuses 1500, 1600 may includeprocessing circuitry 1502 that is capable of executing instructions. Theprocessing circuitry 1502 may include, such as hardware including logiccircuits; a hardware/software combination, such as a processor executingsoftware; or a combination thereof. For example, a processor mayinclude, but is not limited to, a central processing unit (CPU), agraphics processing unit (GPU), an arithmetic logic unit (ALU), adigital signal processor, a microcomputer, a field programmable gatearray (FPGA), a System-on-Chip (SoC), a programmable logic unit, amicroprocessor, application-specific integrated circuit (ASIC), etc.

As further shown in FIGS. 15 and 16 , example apparatuses 1500, 1600 mayinclude a memory 1504 storing instructions 1506. The memory 1504 mayinclude, for example, random-access memory (RAM), read-only memory(ROM), an erasable programmable read-only memory (EPROM), anelectrically erasable programmable read-only memory (EEPROM), etc. Thememory 1504 may be volatile, such as system memory, and/or nonvolatile,such as a hard disk drive, a solid-state storage device, flash memory,or magnetic tape. The instructions 1506 stored in the memory 1504 may bespecified according to a native instruction set architecture of aprocessor, such as a variant of the IA-32 instruction set architectureor a variant of the ARM instruction set architecture, as assembly and/ormachine-language (e.g., binary) instructions; instructions of ahigh-level imperative and/or declarative language that is compilableand/or interpretable to be executed on a processor; and/or instructionsthat are compilable and/or interpretable to be executed by a virtualprocessor of a virtual machine, such as a web browser. A set ofnon-limiting examples of such high-level languages may include, forexample: C, C++, C #, Objective-C, Swift, Haskell, Go, SQL, R, Lisp,Java®, Fortran, Perl, Pascal, Curl, OCaml, JavaScript®, HTMLS (HypertextMarkup Language 5th revision), Ada, ASP (Active Server Pages), PHP (PHP:Hypertext Preprocessor), Scala, Swift, Eiffel, Smalltalk, Erlang, Ruby,Flash®, Visual Basic®, Lua, MATLAB, SIMULINK, and Python®. Suchinstructions 1506 may also include instructions for a library, resource,platform, application programming interface (API), or the like that isutilized in determining a clinical value (such as a prognosis) for anindividual based on a tumor shown in an image.

As shown in FIGS. 15 and 16 , example systems 1508, 1602 may beorganized in a particular manner, for example, to allocate somefunctionality to each component of a system. Some example embodimentsmay implement each such component in various ways, such as software,hardware (e.g., processing circuitry), or a combination thereof. In someexample embodiments, the organization of the system may vary as comparedwith some other example embodiments, including the example systems 1508,1602 shown in FIGS. 15 and 16 . For example, some example embodimentsmay include a system featuring a different organization of components,such as renaming, rearranging, adding, partitioning, duplicating,merging, and/or removing components, sets of components, andrelationships thereamong, without departing from the scope of thepresent disclosure. All such variations that are reasonably technicallyand logically possible, and that are not contradictory with otherstatements, are intended to be included in this disclosure, the scope ofwhich is to be understood as being limited only by the claims.

FIG. 17 is an illustration of an example computer-readable medium 1700,in accordance with some example embodiments.

As shown in FIG. 17 , the non-transitory computer-readable medium 1700may store binary data 1702 encoding a set of instructions 1704 that,when executed by processing circuitry 1502 of an example apparatus 1500,1600, cause the example apparatus 1500, 1600 to determine a clinicalvalue (such as a prognosis) for an individual based on a tumor shown inan image in accordance with some example embodiments. As a first suchexample, the instructions 1704 may encode the elements of an examplemethod 1706, such as the first example method 1300 of FIG. 13 . As asecond such example, the instructions 1704 may encode the elements ofthe second example method 1400 of FIG. 14 . As a third such example, theinstructions 1704 may encode the components of the first example system1508 of FIG. 15 . As a fourth such example, the instructions 1704 mayencode the components of the second example system 1602 of FIG. 16 .

In some example embodiments, a system may include image evaluating meansfor determining a lymphocyte distribution of lymphocytes in an image.For example, the image evaluating means may be or may include one ormore convolutional neural networks and/or any of the other imageevaluation models discussed herein. The system may include classifyingmeans for classifying tumors into a class selected from at least twoclasses respectively associated with lymphocyte distributions. Forexample, the classifying means may be or may include one or moreGaussian mixture models and/or any of the other classifiers discussedherein.

The system may include a tumor evaluator means for determining aclinical value (such as a prognosis) for an individual based on a tumorin an image by invoking the image evaluating means with the image todetermine the lymphocyte distribution of lymphocytes in the tumor,invoking the classifier to classify the tumor into a class based on thelymphocyte distribution, and outputting a clinical value (such as aprognosis) for the individual based on prognoses of individuals withtumors in the class into which the classifier classified the tumor. Forexample, the tumor evaluator means may be or may include a classifier,such as a neural network, a soft- or hard-margin support vector machine,and/or any of the other classifiers discussed herein. For example, thetumor evaluator means may be or may include a display device such as aliquid crystal display (LCD), light-emitting diode (LED), or organiclight-emitting diode (OLED) display; a communication interface such as awebserver, an email server, or a text messaging server; and/or any otheroutput device disclosed herein.

J. Variations

Some example embodiments of the present disclosure may includevariations in many aspects, and some variations may present additionaladvantages and/or reduce disadvantages with respect to other variationsof these sand other techniques. Moreover, some variations may beimplemented in combination, and some combinations may feature additionaladvantages and/or reduced disadvantages through synergistic cooperation.The variations may be incorporated in some example embodiments (e.g.,the first example method of FIG. 13 , the second example method of FIG.14 , the example apparatuses 1500, 1600 and example systems 1508, 1602of FIGS. 15 and 16 , and/or the example non-transitory computer-readablemedium 1700 of FIG. 17 ) to confer individual and/or synergisticadvantages upon such example embodiments.

F1. Scenarios

Some example embodiments may be utilized in a variety of scenarios thatinvolve an analysis of input using distribution-based machine learningmodels. For example, some example embodiments may use the disclosedtechniques to classify tumors for endeavors in various fields of lifesciences, including healthcare and biomedical research. The tumorclassification techniques disclosed herein may be applicable to a widevariety of cancer types, including (without limitation) lung cancertumors, pancreatic adenocarcinoma tumors, and/or breast cancer tumors. Aclinical pathology laboratory may use such techniques to determine tumorclasses of tumor samples, and/or to compare or validate determinationsof tumor classes by individuals and/or other automated processes. Aresearcher may use such techniques to determine tumor classes of tumorsin images of a research data set, which may be from human patients orfrom human or non-human experimental subjects, where such research mayinvolve further techniques for classifying tumors, identifying theprevalence and classes of tumors in different demographics, identifyingrisk factors that are correlated with tumors of different tumor classes,projecting survivability, for determining or comparing the effectivenessof treatment options. A clinician may use the results of theclassification to evaluate the diagnostic, prognostic, and/or treatmentoptions for an individual with a tumor, and/or to explore and understandthe correlation of various risk factors with different tumor classes andthe prognoses of individuals with such tumors. Many such scenarios maybe devised in which the disclosed techniques may be utilized.

F2. Determining Feature Presence and Distribution

In some example embodiments, machine learning models, including deeplearning models, may be used for the detection of various features ofvarious inputs. In various example embodiments, such machine learningmodels may be used to determine the feature map 118 of an image 102, forexample, by generating and applying a mask set 300 of masks 302; todetermine a distribution of the features, such as clusters 204; toperform a classification 402 of areas of an image, such as area typesbased on anatomic features and/or tissue types; to perform a measurement304 of a feature, such as a concentration (e.g., percentage area of anentire image) of a feature or an area type, for example, a lymphocytedensity range estimation 404, using techniques such as binning; togenerate a density map, such as a lymphocyte density map 406; to choosea set of features to be used to classify images 102 or a particularimage 102, such as performing the selection process 700 of FIG. 7 toselect a feature subset; to classify an image 102 of a tumor based on animage feature set or image feature subset; to select a clinical featuresubset from a the values of clinical features 706 of a clinical featureset for an individual or a tumor; to determine the values of clinicalfeatures 706 of a clinical feature set for an individual and/or a tumor;to determine a class of a tumor, such as by preparing and applying a Coxproportional hazards model to clinical features of the tumor; todetermine a class of a tumor based on image features of the tumor (suchas the output of a Gaussian mixture model) and/or clinical features ofthe tumor or the individual (such as the output of a Cox proportionalhazards model); to project a survivability for an individual based on aclassification of a tumor of the individual; and/or to generate one ormore outputs, including visualizations, of such determinations. Each ofthese features and other features of some example embodiments may beperformed, for example, by a machine learning model; by a plurality ofmachine learning models of a same or similar type, such as randomforests, or convolutional neural networks that evaluate different partsof an image or that perform different tasks on an image; and/or by acombination of machine learning models of different types. As one suchexample, in a boosting ensemble, a first machine learning model performsclassification based on the output of other machine learning models.

As a first such example, the presence of the feature (e.g., anactivation within a feature map, and/or a biological activation oflymphocytes) may be determined in various ways. For example, where aninput further includes an image 102, the determining of the presence ofthe feature may be performed by applying at least one convolutionalneural network 110 to the image 102 and receiving, from the at least oneconvolution neural network 110, a feature map 118 indicating thepresence of the feature of the input. That is, a convolutional neuralnetwork 110 may be applied to an image 102 to identify clusters ofpixels 108 in which a feature is apparent. For example, cell-countingconvolutional neural networks 110 may be applied to count cells in atissue sample, where such cells may be lymphocytes. In such scenarios,the tissue sample may be subjected to an assay, such as a dye or aluminescent (such as fluorescent) agent, and a collection of images 102of the tissue sample may be selective for the cells and may thereforenot include other visible components of the tissue sample. The image 102of the tissue sample may then be subjected to a machine learning model(such as a convolutional neural network 110) that may be configured(e.g., trained) to detect shapes such as circles that are indicative ofthe selected cells, and may output a count in the image 102 and/or fordifferent areas of the image 102. Notably, the convolutional neuralnetwork 110 in this case may not be configured and/or used to furtherdetect an arrangement of such features, for example, a number,orientation, and/or positioning of lymphocytes with respect to otherlymphocytes; rather, the counts of respective portions of the image 102may be compiled into a distribution map that may be processed togetherwith a tumor mask to determine the distribution of lymphocytes as beingtumor-invasive, tumor-adjacent, or elsewhere in the tissue sample. Someexample embodiments may use a machine learning model other than aconvolutional neural network to detect the presence of a feature, suchas (e.g.) a non-convolutional neural network such as a fully-connectednetwork or a perceptron network or a Bayesian classifier.

As a second such example, the distribution of the feature may bedetermined in a variety of ways. As a first example, where the inputfurther includes an image 102 illustrating a tissue region of anindividual, determining the distribution of the feature further mayinclude determining an area type (e.g., tumor or non-tumor) for eacharea of the image, and determining the distribution based on the areatype of each area of the image. The distribution of detectedlymphocytes, including lymphocyte counts, may then be determined basedupon the types of tissue in which such counts occur. That is, thedistribution may be determined by tabulating counts of lymphocytes fortumor areas, tumor-adjacent areas (such as stroma), and non-tumor areasof the image 102. As another such example, the determining may includedetermining a boundary of the tissue region within the image 102, anddetermining the distribution based on the boundary of the tissue regionwithin the image 102. That is, the boundaries of the areas of the image102 that are classified as tumor may be determined (e.g., by aconvolutional neural network 110 and/or a human), and an exampleembodiment may tabulate the counts of lymphocytes for all of the areasof the image that are within the determined boundaries of the tumor. Asyet another example, the tissue region of the image including at leasttwo areas, and determining the distribution may include determining acount of lymphocytes within each area of the tissue region anddetermining the distribution based on the count within each area of thetissue region. For example, determining the count within each tissueregion may include determining a density of the count of lymphocyteswithin each area, and then determining the distribution based on thecount within each area. An example is shown in FIG. 3 , in which a firstconvolutional neural network 110-1 is provided to classify areas astumor vs. non-tumor areas and a second convolutional neural network110-2 is provided to estimate a density range of lymphocytes.

As a third such example, the processing of an image 102 to determine thepresence of a feature and/or the distribution of a feature may occur inseveral ways. For example, some example embodiments may be configured topartition an image 102 into a set of areas of the same or varying sizesand/or shapes, such as based on a number of pixels or a correspondingphysical size (e.g., 100-micrometer square areas), and/or based onsimilarity grouping (e.g., identifying areas of similar appearancewithin the image 102). Some example embodiments may be configured toclassify each area (for example, as tumor, tumor-adjacent, ornon-tumor), and/or to determine the distribution by tabulating thepresence of the feature (e.g., a count) within each area of a certainarea type to determine the distribution of lymphocytes. Alternatively, acounting process may be applied to each area, and each area may beclassified based on a count (e.g., high-lymphocyte vs. low-lymphocyteareas). As yet another example, the distributions may be determined in aparametric manner, such as according to a selected distribution type orkernel that a machine learning model may fit to the distribution of thefeature in the input (e.g., a Gaussian mixture model may be applied todetermine Gaussian distributions of subsets of the feature). Otherdistribution models may be applied, including parametric distributionmodels such as chi-square fit, a Poisson distribution, and a betadistribution and non-parametric distribution models such as histograms,binning, and kernel methods.

As a fourth such example, many forms of classifiers may be used, such asBayesian (including naïve Bayesian) classifiers; Gaussian classifiers;probabilistic classifiers; principal component analysis (PCA)classifiers; linear discriminant analysis (LDA) classifiers; quadraticdiscriminant analysis (QDA) classifiers; single-layer or multiplayerperceptron networks; convolutional neural networks; recurrent neuralnetworks; nearest-neighbor classifiers; linear SVM classifiers;radial-basis-function kernel (RBF) SVM classifiers; Gaussian processclassifiers; decision tree classifiers, including random forestclassifiers; and/or restricted or unrestricted Boltzmann machines, amongothers. Examples of convolutional neural network classifiers include,without limitation, LeNet, ZfNet, AlexNet, BN-Inception,CaffeResNet-101, DenseNet-121, DenseNet-169, DenseNet-201, DenseNet-161,DPN-68, DPN-98, DPN-131, FBResNet-152, GoogLeNet, Inception-ResNet-v2,Inception-v3, Inception-v4, MobileNet-v1, MobileNet-v2, NASNet-A-Large,NASNet-A-Mobile, ResNet-101, ResNet-152, ResNet-18, ResNet-34,ResNet-50, ResNext-101, SE-ResNet-101, SE-ResNet-152, SE-ResNet-50,SE-ResNeXt-101, SE-ResNeXt-50, SENet-154, ShuffleNet, SqueezeNet-v1.0,SqueezeNet-v1.1, VGG-11, VGG-11_BN, VGG-13, VGG-13_BN, VGG-16,VGG-16_BN, VGG-19, VGG-19 BN, Xception, DelugeNet, FractalNet,WideResNet, PolyNet, PyramidalNet, and U-net.

In some example embodiments, classification may include regression, andthe term “classification” as used herein is intended to include someexample embodiments that perform regression as an alternative to, or inaddition to, a selection of a class. For example, some exampleembodiments may feature regression as an alternative to or additional toclassification. As a first such example, a determination of a presenceof a feature may include a regression of the presence of the feature,for example, a numerical value indicating a density of the feature in aninput. As a second such example, a determination of a distribution of afeature may include a regression of the distribution of the feature,such as a variance of the regression-based density determined for theinput. As a third such example, the choosing may include performing aregression of the distribution of the feature and choosing a regressionvalue for the distribution of the feature. Such regression aspects maybe performed instead of classification or in addition to aclassification (for example, determining both a presence of the featureand a density of the feature in an area of an image). Some exampleembodiments may involve regression-based machine learning models, suchas Bayesian linear or nonlinear regression, regression-based artificialneural networks such as convolutional neural network regression, supportvector regression, and/or decision tree regression.

Each classifier may be linear or nonlinear; for example, a nonlinearclassifier may be provided (e.g., trained) to perform a linearclassification based upon a kernel transform in a nonlinear space, thatis, a transformation of linear values of a feature vector into nonlinearfeatures. The classifiers may include a variety of techniques to promoteaccurate generalization and classification, such as input normalization,weight regularization, and/or output processing, such as a softmaxactivation output. The classifiers may use a variety of techniques topromote efficient training and/or classification. For example, a two-wayGaussian mixture model may be used in which a same size of the Gaussiandistributions is selected for each dimension of the feature space, whichmay reduce the search space as compared with other Gaussian mixturemodels in which the sizes of the distribution for different dimensionsof the feature space may vary.

Each classifier may be trained to perform classification in a particularmanner, such as supervised learning, unsupervised learning, and/orreinforcement learning. Some example embodiments may include additionaltraining techniques to promote generalization, accuracy, and/orconvergence, such as validation, training data augmentation, and/ordropout regularization. Ensembles of such classifiers may also beutilized, where such ensembles may be homogeneous or heterogeneous, andwherein the classification 122 based on the outputs of the classifiersmay be produced in various ways, such as by consensus, based on theconfidence of each output (e.g., as a weighted combination), and/or viaa stacking architecture such as based on one or more blenders. Theensembles may be trained independently (e.g., a bootstrap aggregationtraining model, or a random forest training model) and/or in sequence(e.g., a boosting training model, such as Adaboost). As an example of aboosting training model, in some support vector machine ensembles, atleast some of the support vector machines may be trained based on anerror of a previously trained support vector machine; e.g., eachsuccessive support vector machine may be trained particularly upon theinputs of the training data set 100 that were incorrectly classified bypreviously trained support vector machines.

As a fifth such example, various forms of classification 122 may beproduced by the one or more classifiers. For example, a perceptron orbinary classifier may output a value indicating whether an input isclassified as a first class 106 or a second class 106, such as whetheran area of an image 102 is a tumor area or a non-tumor area. As anotherexample, a probabilistic classifier may be configured to output aprobability that the input is classified into each class 106 of theclass set 104. Alternatively or additionally, some example embodimentsmay be configured to determine a probability of classifying the inputinto each class 106 of the class set 104, and to choose, from a classset 104 including at least two classes 106, a class 106 of the input,the choosing being based on probabilities of classifying the input intoeach class 106 of the class set 104. For example, a classifier mayoutput a confidence of the classification 122, e.g., a probability ofclassification error, and/or may refrain from outputting aclassification 122 based upon poor confidence, e.g., a minimum-riskclassifier. For example, in areas of an image 102 that are not clearlyidentifiable as tumor or non-tumor, a classifier may be configured torefrain from classifying the area in order to promote accuracy in thecalculated distribution of lymphocytes in areas that may be identifiedas tumor and non-tumor with acceptable confidence.

As a sixth such example, some example embodiments may be configured toprocess the distribution of a feature with a linear or nonlinearclassifier, and may receive, from the linear or nonlinear classifier, aclassification 122 of the class 106 of the input. For example, a linearor nonlinear classifier may include a support vector machine ensemble ofat least two support vector machines, and some example embodiments maybe configured to receive the classification 122 by receiving, from eachof the at least two support vector machines, a candidate classification122, and to determine the classification 122 based on a consensus of thecandidate classifications 122 among the at least two support vectormachines.

As a seventh such example, some example embodiments may be configured touse a linear or nonlinear classifier (including a set or ensemble ofclassifiers) to perform a classification 122 of an input in a variety ofways. For example, an input may be partitioned into areas, and for eachinput portion, an example embodiment may use a classifier to classifythe input portion according to an input portion type selected from a setof input portion types (e.g., performing a classification 122 ofportions of an image 102 of a tumor as tumor areas vs. non-tumor areas,such as shown in the example of FIG. 3 ). The example embodiment maythen be configured to choose, from a class set including at least twoclasses, a class of the input, the choosing based on the distribution ofthe feature for each input portion of an input portion type and thedistribution of the feature for each input portion type of the set ofinput portion types (e.g., classifying areas of tumor-invasivelymphocytes (TIL), tumor-adjacent lymphocytes such as in stroma,high-activation lymphocyte areas, and/or low-activation lymphocyteareas).

As an eighth such example, some example embodiments may be configured toperform a distribution classification by determining a variance of thedistribution of the feature of the input, e.g., the variance of thedistribution over the areas of an input such as an image 102. Someexample embodiments may then be configured to perform classification 122by choosing, from a class set 104 including at least two classes 106, aclass 106 of the input, the choosing being based on the variance of thedistribution of the feature of the input and the variance ofdistribution of the feature for each class 106 of the class set 104. Forexample, some example embodiments may be configured to determine a classof a tumor based, at least in part, upon the variance of thedistribution of lymphocytes over the different areas of the image.

As a ninth such example, some example embodiments may use differenttraining and/or testing to generate and validate the machine learningmodels. For example, training may be performed using heuristics such asstochastic gradient descent, nonlinear conjugate gradient, or simulatedannealing. Training may be performed offline (e.g., based on a fixedtraining data set 100) or online (e.g., continuous training with newtraining data). Training may be evaluated based on various metrics, suchas perceptron error, Kullback-Leibler (KL) divergence, precision, and/orrecall. Training may be performed for a fixed time (e.g., a selectednumber of epochs or generations), until training fails to yieldadditional improvement, and/or until reaching a point of convergence(for example, when classification accuracy reaches a target threshold).A machine learning model may be tested in various ways, such as k-foldcross-validation, to determine the proficiency of the machine learningmodel on previously unseen data. Many such forms of classification 122,classifiers, training, testing, and validation may be included and usedin some example embodiments.

K. Example Computing Environment

FIG. 18 is an illustration of an example apparatus in which some exampleembodiments may be implemented.

FIG. 18 and the following discussion provide a brief, generaldescription of a suitable computing environment to implement embodimentsof one or more of the provisions set forth herein. The operatingenvironment of FIG. 18 is only one example of a suitable operatingenvironment and is not intended to suggest any limitation as to thescope of use or functionality of the operating environment. Examplecomputing devices include, but are not limited to, personal computers,server computers, hand-held or laptop devices, mobile devices (such asmobile phones, Personal Digital Assistants (PDAs), media players, andthe like), multiprocessor systems, media devices such as televisions,consumer electronics, embedded devices, mini computers, mainframecomputers, distributed computing environments that include any of theabove systems or devices, wearable computing devices (such as glasses,earpieces, wristwatches, rings, pendants, handheld and/or body-mountedcameras, clothing-integrated devices, and implantable devices),autonomous vehicles, extended reality (XR) devices such as augmentedreality (AR) and/or virtual reality (VR) devices, internet-of-things(IoT) devices, and the like.

Some example embodiments may include a combination of components of thesame and/or different types, such as a plurality of processors and/orprocessing cores in a uni-processor or multi-processor computer; two ormore processors operating in tandem, such as a CPU and a GPU; a CPUutilizing an ASIC; and/or software executed by processing circuitry.

Some example embodiments may include components of a single device, sucha computer including one or more CPUs that store, access, and manage thecache. Some example embodiments may include components of multipledevices, such as two or more devices having CPUs that communicate toaccess and/or manage a cache. Some example embodiments may include oneor more components that are included in a server computing device, aserver computer, a series of server computers, server farm, a cloudcomputer, a content platform, a mobile computing device, a smartphone, atablet, or a set-top box. Some example embodiments may includecomponents that communicate directly (e.g., two or more cores of amulti-core processor) and/or indirectly (e.g., via a bus, via over awired or wireless channel or network, and/or via an intermediatecomponent such as a microcontroller or arbiter). Some exampleembodiments may include multiple instances of systems or instances thatare respectively performed by a device or component, where such systemsinstances may execute concurrently, consecutively, and/or in aninterleaved manner. Some example embodiments may feature a distributionof an instance or system over two or more devices or components.

Although not required, some example embodiments are described in thegeneral context of “computer readable instructions” being executed byone or more computing devices. Computer readable instructions may bedistributed via computer readable media (discussed below). Computerreadable instructions may be implemented as program modules, such asfunctions, objects, Application Programming Interfaces (APIs), datastructures, and the like, that perform particular tasks or implementparticular abstract data types. Typically, the functionality of thecomputer readable instructions may be combined or distributed as desiredin various environments.

FIG. 18 illustrates an example of an example apparatus 1800 configuredas, or to include, one or more example embodiments, such as the exampleembodiments provided herein. In one apparatus configuration 1802, theexample apparatus 1800 may include processing circuitry 1502 and memory1804. Depending on the exact configuration and type of computing device,memory 1804 may be volatile (such as RAM, for example), nonvolatile(such as ROM, flash memory, etc., for example) or some combination ofthe two.

In some example embodiments, an example apparatus 1800 may includeadditional features and/or functionality. For example, an exampleapparatus 1800 may also include additional storage (e.g., removableand/or non-removable) including, but not limited to, magnetic storage,optical storage, and the like. Such additional storage is illustrated inFIG. 18 by storage 1806. In some example embodiments, computer-readableinstructions to implement one or more embodiments provided herein may bestored in the memory 1804 and/or the storage 1806.

In some example embodiments, the storage 1806 may be configured to storeother computer readable instructions to implement an operating system,an application program, and the like. Computer-readable instructions maybe loaded in memory 1804 for execution by processing circuitry 1502, forexample. Storage may include volatile and nonvolatile, removable andnon-removable media implemented in any method or technology for storageof information such as computer readable instructions or other data.Storage may include, but is not limited to, RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, Digital Versatile Disks(DVDs) or other optical storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices, or any othermedium which may be used to store the desired information and which canbe accessed by example apparatus 1800. Any such computer storage mediamay be part of example apparatus 1800.

In some example embodiments, an example apparatus 1800 may include inputdevice(s) 1810 such as keyboard, mouse, pen, voice input device, touchinput device, infrared cameras, video input devices, and/or any otherinput device. Output device(s) 1808 such as one or more displays,speakers, printers, and/or any other output device may also be includedin example apparatus 1800. Input device(s) 1810 and output device(s)1808 may be connected to example apparatus 1800 via a wired connection,wireless connection, or any combination thereof. In some exampleembodiments, an input device or an output device from another computingdevice may be used as input device(s) 1810 or output device(s) 1808 forexample apparatus 1800.

In some example embodiments, an example apparatus 1800 may be connectedby various interconnects, such as a bus. Such interconnects may includea Peripheral Component Interconnect (PCI), such as PCI Express, aUniversal Serial Bus (USB), Firewire (IEEE 1394), an optical busstructure, and the like. In other example embodiments, components of anexample apparatus 1800 may be interconnected by a network. For example,memory 1804 may include multiple physical memory units located indifferent physical locations interconnected by a network.

In some example embodiments, an example apparatus 1800 may include oneor more communication device(s) 1812 by which the example apparatus 1800may communicate with other devices. Communication device(s) 1812 mayinclude, for example, a modem, a Network Interface Card (NIC), anintegrated network interface, a radio frequency transmitter/receiver, aninfrared port, a USB connection, or other interfaces for connecting theexample apparatus 1800 to other computing devices, including remotedevices 1816. Communication device(s) 1812 may include a wiredconnection or a wireless connection. Communication device(s) 1812 may beconfigured to transmit and/or receive communication media.

Those skilled in the art will realize that storage devices used to storecomputer readable instructions may be distributed across a network. Forexample, an example apparatus 1800 may communicate with a remote device1816 via a network 1814 to store and/or retrieve computer-readableinstructions to implement one or more example embodiments providedherein. For example, an example apparatus 1800 may be configured toaccess a remote device 1816 to download a part or all of thecomputer-readable instructions for execution. Alternatively, an exampleapparatus 1800 may be configured to download portions of thecomputer-readable instructions as needed, wherein some instructions maybe executed at or by the example apparatus 1800 and some otherinstructions may be executed at or by the remote device 1816.

In this application, including the definitions below, the term “module”or the term “controller” may be replaced with the term “circuit.” Theterm “module” may refer to, be part of, or include processing circuitry1502 (shared, dedicated, or group) that executes code and memoryhardware (shared, dedicated, or group) that stores code executed by theprocessing circuitry 1502.

The module may include one or more interface circuits. In some examples,the interface circuit(s) may implement wired or wireless interfaces thatconnect to a local area network (LAN) or a wireless personal areanetwork (WPAN). Examples of a LAN are Institute of Electrical andElectronics Engineers (IEEE) Standard 802.11-2016 (also known as theWIFI wireless networking standard) and IEEE Standard 802.3-2015 (alsoknown as the ETHERNET wired networking standard). Examples of a WPAN areIEEE Standard 802.15.4 (including the ZIGBEE standard from the ZigBeeAlliance) and, from the Bluetooth Special Interest Group (SIG), theBLUETOOTH wireless networking standard (including Core Specificationversions 3.0, 4.0, 4.1, 4.2, 5.0, and 5.1 from the Bluetooth SIG).

The module may communicate with other modules using the interfacecircuit(s). Although the module may be depicted in the presentdisclosure as logically communicating directly with other modules, invarious implementations the module may actually communicate via acommunications system. The communications system includes physicaland/or virtual networking equipment such as hubs, switches, routers, andgateways. In some implementations, the communications system connects toor traverses a wide area network (WAN) such as the Internet. Forexample, the communications system may include multiple LANs connectedto each other over the Internet or point-to-point leased lines usingtechnologies including Multiprotocol Label Switching (MPLS) and virtualprivate networks (VPNs).

In various implementations, the functionality of the module may bedistributed among multiple modules that are connected via thecommunications system. For example, multiple modules may implement thesame functionality distributed by a load balancing system. In a furtherexample, the functionality of the module may be split between a server(also known as remote, or cloud) module and a client (or, user) module.

The term code, as used above, may include software, firmware, and/ormicrocode, and may refer to programs, routines, functions, classes, datastructures, and/or objects. Shared processing circuitry 1502 mayencompass a single microprocessor that executes some or all code frommultiple modules. Group processing circuitry 1502 may encompass amicroprocessor that, in combination with additional microprocessors,executes some or all code from one or more modules. References tomultiple microprocessors encompass multiple microprocessors on discretedies, multiple microprocessors on a single die, multiple cores of asingle microprocessor, multiple threads of a single microprocessor, or acombination of the above.

Shared memory hardware encompasses a single memory device that storessome or all code from multiple modules. Group memory hardwareencompasses a memory device that, in combination with other memorydevices, stores some or all code from one or more modules.

The term memory hardware is a subset of the term computer-readablemedium. The term computer-readable medium, as used herein, does notencompass transitory electrical or electromagnetic signals propagatingthrough a medium (such as on a carrier wave); the term computer-readablemedium is therefore considered tangible and non-transitory. Non-limitingexamples of a non-transitory computer-readable medium are nonvolatilememory devices (such as a flash memory device, an erasable programmableread-only memory device, or a mask read-only memory device), volatilememory devices (such as a static random access memory device or adynamic random access memory device), magnetic storage media (such as ananalog or digital magnetic tape or a hard disk drive), and opticalstorage media (such as a CD, a DVD, or a Blu-ray Disc).

The example embodiments of apparatuses and methods described herein maybe partially or fully implemented by a special-purpose computer createdby configuring a general-purpose computer to execute one or moreparticular functions embodied in computer programs. The functionalblocks and flowchart elements described herein may serve as softwarespecifications, which may be translated into the computer programs bythe routine work of a skilled technician or programmer.

The computer programs include processor-executable instructions that arestored on at least one non-transitory computer-readable medium. Thecomputer programs may also include or rely on stored data. The computerprograms may encompass a basic input/output system (BIOS) that interactswith hardware of the special purpose computer, device drivers thatinteract with particular devices of the special purpose computer, one ormore operating systems, user applications, background services,background applications, etc.

The computer programs may include: (i) descriptive text to be parsed,such as HTML (hypertext markup language), XML (extensible markuplanguage), or JSON (JavaScript Object Notation), (ii) assembly code,(iii) object code generated from source code by a compiler, (iv) sourcecode for execution by an interpreter, (v) source code for compilationand execution by a just-in-time compiler, etc. As examples only, sourcecode may be written using syntax from languages including C, C++, C #,Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl,Pascal, Curl, OCaml, JavaScript®, HTMLS (Hypertext Markup Language 5threvision), Ada, ASP (Active Server Pages), PHP (PHP: HypertextPreprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, VisualBasic®, Lua, MATLAB, SIMULINK, and Python®.

L. Use of Terms

The foregoing description is merely illustrative in nature and is in noway intended to limit the disclosure, its application, or uses. Thebroad teachings of the disclosure can be implemented in a variety offorms. Therefore, while this disclosure includes particular examples,the true scope of the disclosure should not be so limited since othermodifications will become apparent upon a study of the drawings, thespecification, and the following claims. It should be understood thatone or more steps within a method may be executed in different order (orconcurrently) without altering the principles of the present disclosure.Further, although each of the embodiments is described above as havingcertain features, any one or more of those features described withrespect to any embodiment of the disclosure can be implemented in and/orcombined with features of any other example embodiments, even if thatcombination is not explicitly described. In other words, the describedembodiments are not mutually exclusive, and permutations of one or moreembodiments with one another remain within the scope of this disclosure.

Spatial and functional relationships between elements (for example,between modules) are described using various terms, including“connected,” “engaged,” “interfaced,” and “coupled.” Unless explicitlydescribed as being “direct,” when a relationship between first andsecond elements is described in the above disclosure, that relationshipencompasses a direct relationship where no other intervening elementsare present between the first and second elements, and also an indirectrelationship where one or more intervening elements are present (eitherspatially or functionally) between the first and second elements. Asused herein, the phrase at least one of A, B, and C should be construedto mean a logical (A OR B OR C), using a non-exclusive logical OR, andshould not be construed to mean “at least one of A, at least one of B,and at least one of C.”

In the figures, the direction of an arrow, as indicated by thearrowhead, generally demonstrates the flow of information (such as dataor instructions) that is of interest to the illustration. For example,when element A and element B exchange a variety of information butinformation transmitted from element A to element B is relevant to theillustration, the arrow may point from element A to element B. Thisunidirectional arrow does not imply that no other information istransmitted from element B to element A. Further, for information sentfrom element A to element B, element B may send requests for, or receiptacknowledgements of, the information to element A. The term subset doesnot necessarily require a proper subset. In other words, a first subsetof a first set may be coextensive with (equal to) the first set.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

As used herein, the terms “component,” “module,” “system,” “interface,”and the like are generally intended to refer to a computer-relatedentity, either hardware, a combination of hardware and software,software, or software in execution. For example, a component may be, butis not limited to being, a process running on processing circuitry 1502,processing circuitry 1502, an object, an executable, a thread ofexecution, a program, and/or a computer. By way of illustration, both anapplication running on a controller and the controller can be acomponent. One or more components may reside within a process and/orthread of execution and a component may be localized on one computerand/or distributed between two or more computers.

Furthermore, some example embodiments may include a method, apparatus,or article of manufacture using standard programming and/or engineeringtechniques to produce software, firmware, hardware, or any combinationthereof to control a computer to implement the disclosed subject matter.The term “article of manufacture” as used herein is intended toencompass a computer program accessible from any computer-readabledevice, carrier, or media. Of course, those skilled in the art willrecognize many modifications may be made to this configuration withoutdeparting from the scope or spirit of the claimed subject matter.

Various operations of embodiments are provided herein. In some exampleembodiments, one or more of the operations described may constitutecomputer readable instructions stored on one or more computer readablemedia, which if executed by a computing device, will cause the computingdevice to perform the operations described. The order in which some orall of the operations are described should not be construed as to implythat these operations are necessarily order dependent. Alternativeordering will be appreciated by one skilled in the art having thebenefit of this description. Further, it will be understood that not alloperations are necessarily present in each example embodiment providedherein.

As used herein, the term “or” is intended to mean an inclusive “or”rather than an exclusive “or.” That is, unless specified otherwise, orclear from context, “X employs A or B” is intended to mean any of thenatural inclusive permutations. That is, if X employs A; X employs B; orX employs both A and B, then “X employs A or B” is satisfied under anyof the foregoing instances. The articles “a” and “an” as used herein andin the appended claims may generally be construed to mean “one or more”unless specified otherwise or clear from context to be directed to asingular form.

Although the disclosure has been shown and described with respect tosome example embodiments, equivalent alterations and modifications willoccur to others skilled in the art based upon a reading andunderstanding of this specification and the annexed drawings. Thedisclosure includes all such modifications and alterations and islimited only by the scope of the following claims. In particular regardto the various functions performed by the above described components(e.g., elements, resources, etc.), the terms used to describe suchcomponents are intended to correspond, unless otherwise indicated, toany component which performs the specified function of the describedcomponent (e.g., that is functionally equivalent), even though notstructurally equivalent to the disclosed structure which performs thefunction in the herein illustrated some example embodiments of thedisclosure. In addition, while a particular feature of the disclosuremay have been disclosed with respect to only one of severalimplementations, such feature may be combined with one or more otherfeatures of the other implementations as may be desired and advantageousfor any given or particular application. Furthermore, to the extent thatthe terms “includes,” “having,” “has,” “with,” or variants thereof areused in either the detailed description or the claims, such terms areintended to be inclusive in a manner similar to the term “comprising.”

1. A method of operating an apparatus including processing circuitry,the method comprising: executing, by the processing circuitry,instructions that cause the apparatus to: receive an image depicting atleast part of a tumor, determine a lymphocyte distribution oflymphocytes in the tumor based on the image, apply a classifier to thelymphocyte distribution to classify the tumor, the classifier trained toclassify tumors into a class selected from at least two classesassociated with lymphocyte distributions, and determine a clinical valuefor an individual based on a set of prognosis data corresponding toindividuals with tumors in the class into which the classifierclassified the tumor.
 2. The method of claim 1, wherein the tumor is oneof: a pancreatic adenocarcinoma tumor, and a breast cancer tumor.
 3. Themethod of claim 1, wherein: the apparatus further comprises aconvolutional neural network that is trained to determine a lymphocytedistribution of lymphocytes in an area of an image, and the instructionscause the apparatus to invoke the convolutional neural network todetermine the lymphocyte distribution of lymphocytes in respective areasof the image of the tumor.
 4. The method of claim 3, wherein theconvolutional neural network is further trained to classify an area ofthe image as one or more area types selected from an area type setincluding: a tumor area, a lymphocyte area, and a stroma area.
 5. Themethod of claim 4, wherein: determining the lymphocyte distribution oflymphocytes in the tumor includes, for respective lymphocyte areas ofthe image: determining a distance of the lymphocyte area to one or bothof a tumor area or a stroma area, and based on the distance,characterizing the lymphocyte area as one of: a tumor-infiltratinglymphocyte area, a tumor-adjacent lymphocyte area, a stroma-infiltratinglymphocyte area, and a stroma-adjacent lymphocyte area; and theclassifier further classifies the tumor based on the characterizing ofthe lymphocyte area.
 6. The method of claim 4, wherein: determining thelymphocyte distribution of lymphocytes in the tumor includes, forrespective stroma areas of the image, determining a distance of thestroma area to a tumor area, and based on the distance, characterizingthe stroma area as one of: a tumor-infiltrating stroma area, and atumor-adjacent stroma area; and the classifier further classifies thetumor based on the characterizing of the stroma area.
 7. The method ofclaim 1, wherein the at least two classes include: a high-risk class oftumors that are associated with a first survival probability, and alow-risk class of tumors that are associated with a second survivalprobability that is longer than the first survival probability. 8.(canceled)
 9. The method of claim 1, wherein the classifier furthercomprises a Gaussian mixture model configured to determine, forrespective classes, a probability distribution of features for tumors inthe class within a feature space, and the features of the feature spaceof the Gaussian mixture model are selected from a feature set includingat least one of: a measurement of tumor areas of the image, ameasurement of stroma areas of the image, a measurement of lymphocyteareas of the image, a measurement of tumor-infiltrating lymphocyte areasof the image, a measurement of tumor-adjacent lymphocyte areas of theimage, a measurement of stroma-infiltrating lymphocyte areas of theimage, a measurement of stroma-adjacent lymphocyte areas of the image, ameasurement of tumor-infiltrating stroma areas of the image, and ameasurement of tumor-adjacent stroma areas of the image.
 10. The methodof claim 9, wherein, from the feature set, a feature subset is selectedbased on a correlation of the respective classes with respectivefeatures of the subset, and the correlation of the respective classeswith the respective features is based on at least one of a silhouettescore of the feature space and a concordance index.
 11. (canceled) 12.The method of claim 10, wherein the feature subset consists essentiallyof: the measurement of lymphocyte areas of the image, the measurement oftumor-infiltrating lymphocyte areas of the image, the measurement oftumor-adjacent lymphocyte areas of the image, and the measurement oftumor-infiltrating stroma areas of the image.
 13. The method of claim 1,wherein the instructions further cause the apparatus to: apply a Coxproportional hazards model to clinical features of the tumor todetermine a class of the tumor, and determine the clinical value for theindividual based on prognoses of the individuals with tumors in theclass into which the classifier classified the tumor and the classdetermined by the Cox proportional hazards model.
 14. The method ofclaim 13, wherein the clinical features of the tumor of the Coxproportional hazards model are selected from a clinical feature setincluding: a primary diagnosis of the tumor, a location of the tumor, atreatment of the tumor, a measurement of the tumor, a metastaticcondition of the tumor, a primary diagnosis of the individual, aprevious cancer medical history of the individual, a race of theindividual, an ethnicity of the individual, a gender of the individual,a smoking habit frequency of the individual, a smoking habit duration ofthe individual, and an alcohol history of the individual.
 15. The methodof claim 14, wherein, from the clinical feature set, a clinical featuresubset of features is selected for the Cox proportional hazards modelbased on a correlation of the respective classes with respectiveclinical features of the clinical feature subset and the clinicalfeature subset consists of the measurement of the tumor and themetastatic condition of the tumor.
 16. (canceled)
 17. The method ofclaim 1, wherein: the at least two classes are a low-risk tumor classand a high-risk tumor class, determining the lymphocyte distributionfurther comprises applying a convolutional neural network to the image,the convolutional neural network configured to measure the lymphocytedistribution of lymphocytes for different area types of the image, theclassifier is a two-way Gaussian mixture model configured to determine,for respective classes, a probability distribution of features fortumors in the class within a feature space, the method further comprisesapplying a Cox proportional hazards model to clinical features of thetumor to determine a class of the tumor, and determining the clinicalvalue for the individual is further based on the class determined by theCox proportional hazards model.
 18. The method of claim 1, wherein theinstructions further cause the apparatus to display a Kaplan Meiersurvivability projection of the clinical value for the individual. 19.(canceled)
 20. The method of claim 1, wherein the instructions furthercause the apparatus to determine at least one of: a diagnostic test forthe tumor based on the clinical value for the individual: a treatment ofthe individual based on the clinical value for the individual; and aschedule of a therapeutic agent for treating the tumor based on theclinical value for the individual. 21-26. (canceled)
 27. An apparatuscomprising: a memory storing instructions; and processing circuitryconfigured by execution of the instructions stored in the memory todetermine a clinical value for an individual based on a tumor in animage by: determining a lymphocyte distribution of lymphocytes in atumor based on an image of the tumor, applying a classifier to thelymphocyte distribution to classify the tumor, the classifier configuredto classify tumors into a class selected from at least two classesrespectively associated with lymphocyte distributions, and outputtingthe clinical value based on prognoses of individuals with tumors in theclass into which the classifier classified the tumor.
 28. The apparatusof claim 27, wherein: the at least two classes are a low-risk tumorclass and a high-risk tumor class, determining the lymphocytedistribution further comprises applying a convolutional neural networkto the image, the convolutional neural network configured to measure thelymphocyte distribution of lymphocytes for different area types of theimage, the classifier includes a two-way Gaussian mixture modelconfigured to determine, for respective classes, a probabilitydistribution of features for tumors in the class within a feature space,and the instructions further cause the processing circuitry to: apply aCox proportional hazards model to clinical features of the tumor todetermine a class of the tumor, and determine the clinical value for theindividual based on the prognoses of the individuals with tumors in theclass into which the classifier classified the tumor and the classdetermined by the Cox proportional hazards model.
 29. A non-transitorycomputer-readable medium storing instructions that, when executed byprocessing circuitry, cause the processing circuitry to determine aclinical value for an individual based on a tumor in an image by:determining a lymphocyte distribution of lymphocytes in a tumor based onan image of the tumor, applying a classifier to the lymphocytedistribution to classify the tumor, the classifier configured toclassify tumors into a class selected from at least two classesassociated with lymphocyte distributions, and outputting the clinicalvalue for the individual based on prognoses of individuals with tumorsin the class into which the classifier classified the tumor.
 30. Thenon-transitory computer-readable medium of claim 29, wherein: the atleast two classes are a low-risk tumor class and a high-risk tumorclass, determining the lymphocyte distribution further comprisesapplying a convolutional neural network to the image, the convolutionalneural network configured to measure the lymphocyte distribution oflymphocytes for different area types of the image, the classifierincludes a two-way Gaussian mixture model configured to determine, forrespective classes, a probability distribution of features for tumors inthe class within a feature space, and the instructions further cause theprocessing circuitry to: apply a Cox proportional hazards model toclinical features of the tumor to determine a class of the tumor, anddetermine the clinical value for the individual based on the prognosesof the individuals with tumors in the class into which the classifierclassified the tumor and the class determined by the Cox proportionalhazards model.